[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:46:46 dist-test-slave-86rs krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:46:46 dist-test-slave-86rs krb5kdc[399](info): set up 2 sockets
Sep 27 00:46:46 dist-test-slave-86rs krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 00:46:47.194667 392 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:46:47 dist-test-slave-86rs krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:46:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:46:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:46:47.344534 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39545
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:47.481400 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:47.481834 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:47.481921 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:47.486414 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:46:47.486536 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:47.486577 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:47.486613 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:46:47.486663 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:46:47.492875 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:46:47.494475 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:47.496065 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:46:47.507966 412 server_base.cc:1034] running on GCE node
W20260927 00:46:47.508025 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:47.508005 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:47.508291 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:47.509053 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:47.510541 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:47.511744 412 hybrid_clock.cc:648] HybridClock initialized: now 1790470007511683 us; error 80 us; skew 500 ppm
Sep 27 00:46:47 dist-test-slave-86rs krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:47.515130 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:46:47.516383 412 webserver.cc:460] Webserver started at http://127.0.98.62:36127/ using document root <none> and password file <none>
I20260927 00:46:47.517127 412 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:47.517231 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:47.517511 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:46:47.520126 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "385238e988c646d5bcdd77cc1014d071"
format_stamp: "Formatted at 2026-09-27 00:46:47 on dist-test-slave-86rs"
server_key: "832eca2414b2b2fc16f1ba70605a85f3"
server_key_iv: "6b8ad7a98e75a6774017f22dc252d8c3"
server_key_version: "encryptionkey@0"
I20260927 00:46:47.520881 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "385238e988c646d5bcdd77cc1014d071"
format_stamp: "Formatted at 2026-09-27 00:46:47 on dist-test-slave-86rs"
server_key: "832eca2414b2b2fc16f1ba70605a85f3"
server_key_iv: "6b8ad7a98e75a6774017f22dc252d8c3"
server_key_version: "encryptionkey@0"
I20260927 00:46:47.525764 412 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 00:46:47.528843 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:47.530619 412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:46:47.530767 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "385238e988c646d5bcdd77cc1014d071"
format_stamp: "Formatted at 2026-09-27 00:46:47 on dist-test-slave-86rs"
server_key: "832eca2414b2b2fc16f1ba70605a85f3"
server_key_iv: "6b8ad7a98e75a6774017f22dc252d8c3"
server_key_version: "encryptionkey@0"
I20260927 00:46:47.530890 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:47.543193 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:47.547582 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:47.547803 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:47.556797 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39545
I20260927 00:46:47.556834 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39545 every 8 connection(s)
I20260927 00:46:47.558035 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:46:47.562040 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:47.564246 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 412
I20260927 00:46:47.564482 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:46:47.566573 392 external_mini_cluster.cc:1428] Setting key a904e00e3e9898d63cdb905a4a70afd9
I20260927 00:46:47.571266 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 385238e988c646d5bcdd77cc1014d071: Bootstrap starting.
I20260927 00:46:47.575103 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 385238e988c646d5bcdd77cc1014d071: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:47.576336 480 log.cc:824] T 00000000000000000000000000000000 P 385238e988c646d5bcdd77cc1014d071: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:47.579134 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 385238e988c646d5bcdd77cc1014d071: No bootstrap required, opened a new log
I20260927 00:46:47.584939 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 385238e988c646d5bcdd77cc1014d071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385238e988c646d5bcdd77cc1014d071" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39545 } }
I20260927 00:46:47.585393 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 385238e988c646d5bcdd77cc1014d071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:47.585462 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 385238e988c646d5bcdd77cc1014d071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 385238e988c646d5bcdd77cc1014d071, State: Initialized, Role: FOLLOWER
I20260927 00:46:47.586153 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 385238e988c646d5bcdd77cc1014d071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385238e988c646d5bcdd77cc1014d071" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39545 } }
I20260927 00:46:47.586341 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 385238e988c646d5bcdd77cc1014d071 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:46:47.586421 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 385238e988c646d5bcdd77cc1014d071 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:46:47.586556 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 385238e988c646d5bcdd77cc1014d071 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:47.588116 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 385238e988c646d5bcdd77cc1014d071 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385238e988c646d5bcdd77cc1014d071" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39545 } }
I20260927 00:46:47.588666 480 leader_election.cc:302] T 00000000000000000000000000000000 P 385238e988c646d5bcdd77cc1014d071 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 385238e988c646d5bcdd77cc1014d071; no voters:
I20260927 00:46:47.589140 480 leader_election.cc:288] T 00000000000000000000000000000000 P 385238e988c646d5bcdd77cc1014d071 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:46:47.589278 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 385238e988c646d5bcdd77cc1014d071 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:46:47.589577 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 385238e988c646d5bcdd77cc1014d071 [term 1 LEADER]: Becoming Leader. State: Replica: 385238e988c646d5bcdd77cc1014d071, State: Running, Role: LEADER
I20260927 00:46:47.590010 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 385238e988c646d5bcdd77cc1014d071 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385238e988c646d5bcdd77cc1014d071" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39545 } }
I20260927 00:46:47.591221 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 385238e988c646d5bcdd77cc1014d071 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:46:47.592080 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 385238e988c646d5bcdd77cc1014d071 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 385238e988c646d5bcdd77cc1014d071. Latest consensus state: current_term: 1 leader_uuid: "385238e988c646d5bcdd77cc1014d071" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385238e988c646d5bcdd77cc1014d071" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39545 } } }
I20260927 00:46:47.592242 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 385238e988c646d5bcdd77cc1014d071 [sys.catalog]: This master's current role is: LEADER
I20260927 00:46:47.592696 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 385238e988c646d5bcdd77cc1014d071 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "385238e988c646d5bcdd77cc1014d071" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385238e988c646d5bcdd77cc1014d071" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39545 } } }
Sep 27 00:46:47 dist-test-slave-86rs krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:47.592796 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 385238e988c646d5bcdd77cc1014d071 [sys.catalog]: This master's current role is: LEADER
I20260927 00:46:47.593231 490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:46:47.598425 490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:46:47.601269 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:47.571350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59654 (local address 127.0.98.62:39545)
0927 00:46:47.571992 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:47.572016 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:47.573429 (+ 1413us) server_negotiation.cc:407] Connection header received
0927 00:46:47.578652 (+ 5223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:47.578674 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:47.579081 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:47.579466 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:47.583332 (+ 3866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:47.584559 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:47.586599 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:47.587195 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:47.587379 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:47.593934 (+ 6555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:47.593942 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:47.593962 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:47.594000 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:47.596376 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:47.597129 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:47.597137 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:47.597143 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:47.597303 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:47.597721 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:47.597724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:47.597726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:47.598299 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:47.598523 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:47.599661 (+ 1138us) server_negotiation.cc:299] Negotiation successful
0927 00:46:47.600034 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":200,"threads_started":1}
I20260927 00:46:47.606413 490 catalog_manager.cc:1347] Generated new cluster ID: b5340b4d92fc46cc9f0f287fa0dbf4a4
I20260927 00:46:47.606601 490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:46:47.633018 490 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:46:47.634310 490 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:46:47.645895 490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 385238e988c646d5bcdd77cc1014d071: Generated new TSK 0
I20260927 00:46:47.647092 490 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:46:47.690243 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39545
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:47.829329 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:47.829731 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:47.829787 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:47.834504 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:47.834596 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:47.834677 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:46:47.841164 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39545
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:46:47.842701 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:47.844066 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:47.856614 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:47.856621 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:47.856564 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:47.856947 506 server_base.cc:1034] running on GCE node
I20260927 00:46:47.857535 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:47.858419 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:47.859596 506 hybrid_clock.cc:648] HybridClock initialized: now 1790470007859571 us; error 42 us; skew 500 ppm
Sep 27 00:46:47 dist-test-slave-86rs krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:47.863055 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:46:47.864384 506 webserver.cc:460] Webserver started at http://127.0.98.1:33891/ using document root <none> and password file <none>
I20260927 00:46:47.865149 506 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:47.865254 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:47.865545 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:46:47.868290 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-0/data/instance:
uuid: "a1174734dc064795958f6906d7eff26b"
format_stamp: "Formatted at 2026-09-27 00:46:47 on dist-test-slave-86rs"
server_key: "87101b790b8db8436284563462658089"
server_key_iv: "75c042b432b1a26a5f09dcdae39cfe5e"
server_key_version: "encryptionkey@0"
I20260927 00:46:47.868995 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a1174734dc064795958f6906d7eff26b"
format_stamp: "Formatted at 2026-09-27 00:46:47 on dist-test-slave-86rs"
server_key: "87101b790b8db8436284563462658089"
server_key_iv: "75c042b432b1a26a5f09dcdae39cfe5e"
server_key_version: "encryptionkey@0"
I20260927 00:46:47.873734 506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 00:46:47.876865 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:47.878454 506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:46:47.878618 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "a1174734dc064795958f6906d7eff26b"
format_stamp: "Formatted at 2026-09-27 00:46:47 on dist-test-slave-86rs"
server_key: "87101b790b8db8436284563462658089"
server_key_iv: "75c042b432b1a26a5f09dcdae39cfe5e"
server_key_version: "encryptionkey@0"
I20260927 00:46:47.878784 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:47.891495 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:47.895612 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:47.895872 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:47.896759 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:46:47.897819 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:46:47.897891 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:47.897975 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:46:47.898027 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:47.908981 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42565
I20260927 00:46:47.909001 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42565 every 8 connection(s)
I20260927 00:46:47.910295 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:46:47.919301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 506
I20260927 00:46:47.919476 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:46:47.919803 392 external_mini_cluster.cc:1428] Setting key ad3a315321a7926948ae7c1e484faaa3
Sep 27 00:46:47 dist-test-slave-86rs krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:47.929194 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:47.913475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50753 (local address 127.0.98.62:39545)
0927 00:46:47.913645 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:47.913650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:47.914876 (+ 1226us) server_negotiation.cc:407] Connection header received
0927 00:46:47.916646 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:47.916650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:47.916734 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:47.916854 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:47.918443 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:47.919092 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:47.920668 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:47.921000 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:47.921226 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:47.924639 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:47.924643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:47.924645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:47.924676 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:47.926289 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:47.926942 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:47.926946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:47.926947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:47.927034 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:47.927523 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:47.927527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:47.927528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:47.927792 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:47.927879 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:47.928875 (+ 996us) server_negotiation.cc:299] Negotiation successful
0927 00:46:47.929029 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":61}
I20260927 00:46:47.930974 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:47.914090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39545 (local address 127.0.98.1:50753)
0927 00:46:47.914685 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:47.914744 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:47.916357 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:47.917098 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:47.917108 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:47.917593 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:47.918270 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:47.918287 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:47.919249 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:47.919252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:47.920511 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:47.920521 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:47.921174 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:47.921177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:47.921371 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:47.922185 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:47.922228 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:47.924400 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:47.926442 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:47.926450 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:47.926482 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:47.926809 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:47.927136 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:47.927142 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:47.927147 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:47.927395 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:47.927914 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:47.927927 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:47.928505 (+ 578us) client_negotiation.cc:769] Sending connection context
0927 00:46:47.928823 (+ 318us) client_negotiation.cc:240] Negotiation successful
0927 00:46:47.929381 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":144,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:46:47.932802 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39545
I20260927 00:46:47.933166 635 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:47.933912 635 heartbeater.cc:511] Master 127.0.98.62:39545 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:46:47.936329 444 ts_manager.cc:194] Registered new tserver with Master: a1174734dc064795958f6906d7eff26b (127.0.98.1:42565)
I20260927 00:46:47.937865 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:46:47.951681 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39545
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:48.093636 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:48.094156 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:48.094223 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:48.099150 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:48.099282 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:48.099376 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:46:48.106333 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39545
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:46:48.107950 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:48.109511 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:48.122552 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:48.122853 640 server_base.cc:1034] running on GCE node
W20260927 00:46:48.122552 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:48.122586 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:48.123646 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:48.124691 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:48.125895 640 hybrid_clock.cc:648] HybridClock initialized: now 1790470008125866 us; error 50 us; skew 500 ppm
Sep 27 00:46:48 dist-test-slave-86rs krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:48.129841 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:46:48.131137 640 webserver.cc:460] Webserver started at http://127.0.98.2:36905/ using document root <none> and password file <none>
I20260927 00:46:48.132036 640 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:48.132164 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:48.132530 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:46:48.135519 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-1/data/instance:
uuid: "7c15669afe4c4980b8668c6e0dd2738d"
format_stamp: "Formatted at 2026-09-27 00:46:48 on dist-test-slave-86rs"
server_key: "1b8438f767a1204978dfae71fb2799bf"
server_key_iv: "552e8445b02442cd72153cf47cb41a0b"
server_key_version: "encryptionkey@0"
I20260927 00:46:48.136327 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7c15669afe4c4980b8668c6e0dd2738d"
format_stamp: "Formatted at 2026-09-27 00:46:48 on dist-test-slave-86rs"
server_key: "1b8438f767a1204978dfae71fb2799bf"
server_key_iv: "552e8445b02442cd72153cf47cb41a0b"
server_key_version: "encryptionkey@0"
I20260927 00:46:48.141355 640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 00:46:48.145049 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:48.147080 640 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:46:48.147284 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "7c15669afe4c4980b8668c6e0dd2738d"
format_stamp: "Formatted at 2026-09-27 00:46:48 on dist-test-slave-86rs"
server_key: "1b8438f767a1204978dfae71fb2799bf"
server_key_iv: "552e8445b02442cd72153cf47cb41a0b"
server_key_version: "encryptionkey@0"
I20260927 00:46:48.147467 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:48.171295 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:48.176200 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:48.176527 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:48.177399 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:46:48.178522 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:46:48.178603 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:48.178694 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:46:48.178750 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:48.192510 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35415
I20260927 00:46:48.192543 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35415 every 8 connection(s)
I20260927 00:46:48.193938 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:46:48.200771 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 640
I20260927 00:46:48.200898 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:46:48.201192 392 external_mini_cluster.cc:1428] Setting key 31ae12dd4d8b0a6352f5845bd10db395
Sep 27 00:46:48 dist-test-slave-86rs krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:46:48.214044 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:48.196789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40643 (local address 127.0.98.62:39545)
0927 00:46:48.196984 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:48.196988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:48.198144 (+ 1156us) server_negotiation.cc:407] Connection header received
0927 00:46:48.199926 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:48.199930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:48.200003 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:48.200097 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:48.202090 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:48.202724 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:48.204442 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:48.204873 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:48.205005 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:48.208801 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:48.208806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:48.208808 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:48.208844 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:48.210732 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:48.211478 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:48.211481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:48.211483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:48.211582 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:48.212035 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:48.212040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:48.212044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:48.212363 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:48.212491 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:48.213449 (+ 958us) server_negotiation.cc:299] Negotiation successful
0927 00:46:48.213729 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 00:46:48.215474 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:48.197339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39545 (local address 127.0.98.2:40643)
0927 00:46:48.197942 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:48.198000 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:48.199623 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:48.200413 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:48.200424 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:48.201051 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:48.201927 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:48.201951 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:48.202935 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:48.202940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:48.204212 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:48.204310 (+ 98us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:48.205125 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:48.205132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:48.205458 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:48.206312 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:48.206344 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:48.208525 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:48.210910 (+ 2385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:48.210919 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:48.210955 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:48.211314 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:48.211700 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:48.211704 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:48.211707 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:48.211908 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:48.212488 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:48.212499 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:48.213009 (+ 510us) client_negotiation.cc:769] Sending connection context
0927 00:46:48.213330 (+ 321us) client_negotiation.cc:240] Negotiation successful
0927 00:46:48.213858 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":154,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:46:48.217546 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39545
I20260927 00:46:48.217922 769 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:48.218767 769 heartbeater.cc:511] Master 127.0.98.62:39545 requested a full tablet report, sending...
I20260927 00:46:48.220311 444 ts_manager.cc:194] Registered new tserver with Master: 7c15669afe4c4980b8668c6e0dd2738d (127.0.98.2:35415)
I20260927 00:46:48.221340 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:46:48.235680 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39545
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:48.376878 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:48.377264 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:48.377317 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:48.381783 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:48.381893 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:48.381970 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:46:48.388692 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39545
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:46:48.390241 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:48.391633 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:48.404116 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:48.404116 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:48.404129 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:48.404632 774 server_base.cc:1034] running on GCE node
I20260927 00:46:48.405205 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:48.406050 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:48.407262 774 hybrid_clock.cc:648] HybridClock initialized: now 1790470008407247 us; error 45 us; skew 500 ppm
Sep 27 00:46:48 dist-test-slave-86rs krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:48.416308 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:46:48.417630 774 webserver.cc:460] Webserver started at http://127.0.98.3:38769/ using document root <none> and password file <none>
I20260927 00:46:48.418438 774 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:48.418545 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:48.418843 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:46:48.421698 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-2/data/instance:
uuid: "a63b020e0b3a45f0868ebd38dcf1ad5c"
format_stamp: "Formatted at 2026-09-27 00:46:48 on dist-test-slave-86rs"
server_key: "c84d0c4b7293a3823a5a5d1edc9993fb"
server_key_iv: "223c9fc037f1ad89362abe96e68a5177"
server_key_version: "encryptionkey@0"
I20260927 00:46:48.422474 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a63b020e0b3a45f0868ebd38dcf1ad5c"
format_stamp: "Formatted at 2026-09-27 00:46:48 on dist-test-slave-86rs"
server_key: "c84d0c4b7293a3823a5a5d1edc9993fb"
server_key_iv: "223c9fc037f1ad89362abe96e68a5177"
server_key_version: "encryptionkey@0"
I20260927 00:46:48.427685 774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 00:46:48.431123 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:48.433108 774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 00:46:48.433296 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "a63b020e0b3a45f0868ebd38dcf1ad5c"
format_stamp: "Formatted at 2026-09-27 00:46:48 on dist-test-slave-86rs"
server_key: "c84d0c4b7293a3823a5a5d1edc9993fb"
server_key_iv: "223c9fc037f1ad89362abe96e68a5177"
server_key_version: "encryptionkey@0"
I20260927 00:46:48.433468 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:48.461225 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:48.465647 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:48.465898 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:48.466698 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:46:48.467764 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:46:48.467839 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:48.467916 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:46:48.467960 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:48.480017 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41417
I20260927 00:46:48.480037 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41417 every 8 connection(s)
I20260927 00:46:48.481249 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:46:48.485291 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 774
I20260927 00:46:48.485435 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790470006085796-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:46:48.485757 392 external_mini_cluster.cc:1428] Setting key e267266158b989a810707734f6b3b9d1
Sep 27 00:46:48 dist-test-slave-86rs krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:48.499296 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:48.483713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37015 (local address 127.0.98.62:39545)
0927 00:46:48.483901 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:48.483904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:48.485210 (+ 1306us) server_negotiation.cc:407] Connection header received
0927 00:46:48.486946 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:48.486950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:48.487019 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:48.487136 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:48.489724 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:48.490328 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:48.491775 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:48.492019 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:48.492194 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:48.495290 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:48.495294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:48.495296 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:48.495328 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:48.497048 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:48.497591 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:48.497595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:48.497596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:48.497681 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:48.498064 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:48.498067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:48.498069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:48.498297 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:48.498398 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:48.499004 (+ 606us) server_negotiation.cc:299] Negotiation successful
0927 00:46:48.499139 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 00:46:48.500651 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:48.484343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39545 (local address 127.0.98.3:37015)
0927 00:46:48.484957 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:48.485031 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:48.486679 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:48.487381 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:48.487391 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:48.487847 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:48.489553 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:48.489577 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:48.490468 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:48.490472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:48.491642 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:48.491652 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:48.492305 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:48.492310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:48.492492 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:48.493335 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:48.493355 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:48.495057 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:48.497157 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:48.497163 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:48.497181 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:48.497478 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:48.497761 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:48.497764 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:48.497765 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:48.497915 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:48.498403 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:48.498410 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:48.498721 (+ 311us) client_negotiation.cc:769] Sending connection context
0927 00:46:48.498909 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 00:46:48.499261 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":147,"threads_started":1}
I20260927 00:46:48.501950 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39545
I20260927 00:46:48.502286 903 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:48.502909 903 heartbeater.cc:511] Master 127.0.98.62:39545 requested a full tablet report, sending...
I20260927 00:46:48.504093 443 ts_manager.cc:194] Registered new tserver with Master: a63b020e0b3a45f0868ebd38dcf1ad5c (127.0.98.3:41417)
I20260927 00:46:48.504971 443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37015
I20260927 00:46:48.514163 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 00:46:48 dist-test-slave-86rs krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470008, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:46:48 dist-test-slave-86rs krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470008, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:46:48.545729 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:48.534484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52430 (local address 127.0.98.1:42565)
0927 00:46:48.534905 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:48.534922 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:48.534955 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:46:48.535074 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:48.535081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:48.535444 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:48.535605 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:48.536515 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:48.537420 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:48.538320 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:48.538870 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:48.538965 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:48.541865 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:48.541876 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:48.541895 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:48.541925 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:48.543617 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:48.544103 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:48.544110 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:48.544112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:48.544213 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:48.544567 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:48.544572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:48.544574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:48.544889 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:48.545157 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:48.545324 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 00:46:48.545520 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":186,"threads_started":1}
W20260927 00:46:48.547106 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52430
Sep 27 00:46:48 dist-test-slave-86rs krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:46:48 dist-test-slave-86rs krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:46:48.580124 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:48.568182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52432 (local address 127.0.98.1:42565)
0927 00:46:48.568409 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:48.568415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:48.568536 (+ 121us) server_negotiation.cc:407] Connection header received
0927 00:46:48.568755 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:48.568757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:48.568815 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:48.568907 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:48.569794 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:48.570489 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:48.571696 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:48.571946 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:48.572062 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:48.576091 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:48.576096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:48.576099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:48.576136 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:48.578100 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:48.578759 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:48.578764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:48.578766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:48.578865 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:48.579192 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:48.579195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:48.579197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:48.579437 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:48.579613 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:48.579782 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 00:46:48.579930 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260927 00:46:48.581416 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 506
I20260927 00:46:48.588128 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 640
I20260927 00:46:48.597895 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 774
I20260927 00:46:48.604668 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 412
2026-09-27T00:46:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2506 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:46:48 dist-test-slave-86rs krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:46:48 dist-test-slave-86rs krb5kdc[917](info): set up 2 sockets
Sep 27 00:46:48 dist-test-slave-86rs krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260927 00:46:49.688567 392 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260927 00:46:49.980068 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.190s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/test-user.keytab.
W20260927 00:46:50.083587 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.103s user 0.001s sys 0.001s
Sep 27 00:46:50 dist-test-slave-86rs krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:46:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:46:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:46:50.144001 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38767
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:50.283643 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:50.284060 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:50.284112 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:50.288476 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:46:50.288568 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:50.288592 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:50.288614 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:46:50.288632 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:46:50.294526 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:46:50.295967 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:50.297410 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:50.308422 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:50.308422 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:50.308422 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:50.308847 930 server_base.cc:1034] running on GCE node
I20260927 00:46:50.309634 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:50.311209 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:50.312384 930 hybrid_clock.cc:648] HybridClock initialized: now 1790470010312336 us; error 56 us; skew 500 ppm
Sep 27 00:46:50 dist-test-slave-86rs krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:50.316051 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:46:50.317446 930 webserver.cc:460] Webserver started at http://127.0.98.62:46611/ using document root <none> and password file <none>
I20260927 00:46:50.318253 930 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:50.318362 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:50.318666 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:46:50.321694 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "49731b3f4af54212893befcb808514a4"
format_stamp: "Formatted at 2026-09-27 00:46:50 on dist-test-slave-86rs"
server_key: "32b9ad1623bdab358161ee70258b893a"
server_key_iv: "4ad9cc8c943f6869791d4ee9c7161530"
server_key_version: "encryptionkey@0"
I20260927 00:46:50.322603 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "49731b3f4af54212893befcb808514a4"
format_stamp: "Formatted at 2026-09-27 00:46:50 on dist-test-slave-86rs"
server_key: "32b9ad1623bdab358161ee70258b893a"
server_key_iv: "4ad9cc8c943f6869791d4ee9c7161530"
server_key_version: "encryptionkey@0"
I20260927 00:46:50.327965 930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 00:46:50.331446 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:50.333530 930 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:46:50.333722 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "49731b3f4af54212893befcb808514a4"
format_stamp: "Formatted at 2026-09-27 00:46:50 on dist-test-slave-86rs"
server_key: "32b9ad1623bdab358161ee70258b893a"
server_key_iv: "4ad9cc8c943f6869791d4ee9c7161530"
server_key_version: "encryptionkey@0"
I20260927 00:46:50.333884 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:50.358315 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:50.362723 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:50.362984 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:50.372176 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38767
I20260927 00:46:50.372188 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38767 every 8 connection(s)
I20260927 00:46:50.373457 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:46:50.373677 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 930
I20260927 00:46:50.373816 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:46:50.374133 392 external_mini_cluster.cc:1428] Setting key 1893873c0997811fab4bc45a0fa1a310
I20260927 00:46:50.377456 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 00:46:50 dist-test-slave-86rs krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:50.385190 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49731b3f4af54212893befcb808514a4: Bootstrap starting.
I20260927 00:46:50.388594 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49731b3f4af54212893befcb808514a4: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:50.389608 998 log.cc:824] T 00000000000000000000000000000000 P 49731b3f4af54212893befcb808514a4: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:50.390909 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:50.375996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52892 (local address 127.0.98.62:38767)
0927 00:46:50.376505 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:50.376537 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:50.376583 (+ 46us) server_negotiation.cc:407] Connection header received
0927 00:46:50.377376 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:50.377398 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:50.377891 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:50.378329 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:50.379243 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.380447 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.381385 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.381946 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.382132 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:50.385219 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:50.385225 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:50.385244 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:50.385302 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:50.387181 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:50.387733 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:50.387739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:50.387745 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:50.387878 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:50.388196 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:50.388200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:50.388203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:50.388810 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:50.389046 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:50.389355 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 00:46:50.389749 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":197,"threads_started":1}
I20260927 00:46:50.392279 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49731b3f4af54212893befcb808514a4: No bootstrap required, opened a new log
I20260927 00:46:50.396822 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49731b3f4af54212893befcb808514a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49731b3f4af54212893befcb808514a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38767 } }
I20260927 00:46:50.397181 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49731b3f4af54212893befcb808514a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:50.397250 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49731b3f4af54212893befcb808514a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49731b3f4af54212893befcb808514a4, State: Initialized, Role: FOLLOWER
I20260927 00:46:50.397835 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49731b3f4af54212893befcb808514a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49731b3f4af54212893befcb808514a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38767 } }
I20260927 00:46:50.398023 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49731b3f4af54212893befcb808514a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:46:50.398099 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49731b3f4af54212893befcb808514a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:46:50.398216 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49731b3f4af54212893befcb808514a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:50.399410 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49731b3f4af54212893befcb808514a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49731b3f4af54212893befcb808514a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38767 } }
I20260927 00:46:50.399863 998 leader_election.cc:302] T 00000000000000000000000000000000 P 49731b3f4af54212893befcb808514a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49731b3f4af54212893befcb808514a4; no voters:
I20260927 00:46:50.400338 998 leader_election.cc:288] T 00000000000000000000000000000000 P 49731b3f4af54212893befcb808514a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:46:50.400473 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49731b3f4af54212893befcb808514a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:46:50.400750 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49731b3f4af54212893befcb808514a4 [term 1 LEADER]: Becoming Leader. State: Replica: 49731b3f4af54212893befcb808514a4, State: Running, Role: LEADER
I20260927 00:46:50.401118 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49731b3f4af54212893befcb808514a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49731b3f4af54212893befcb808514a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38767 } }
I20260927 00:46:50.402011 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49731b3f4af54212893befcb808514a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:46:50.403373 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49731b3f4af54212893befcb808514a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49731b3f4af54212893befcb808514a4. Latest consensus state: current_term: 1 leader_uuid: "49731b3f4af54212893befcb808514a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49731b3f4af54212893befcb808514a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38767 } } }
I20260927 00:46:50.403520 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49731b3f4af54212893befcb808514a4 [sys.catalog]: This master's current role is: LEADER
I20260927 00:46:50.403592 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49731b3f4af54212893befcb808514a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49731b3f4af54212893befcb808514a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49731b3f4af54212893befcb808514a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38767 } } }
I20260927 00:46:50.403683 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49731b3f4af54212893befcb808514a4 [sys.catalog]: This master's current role is: LEADER
I20260927 00:46:50.404109 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:46:50.408360 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:46:50.416080 1012 catalog_manager.cc:1347] Generated new cluster ID: 46b5a28e2256473a8794909f39ab000d
I20260927 00:46:50.416216 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:46:50.423026 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:46:50.424278 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:46:50.436520 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49731b3f4af54212893befcb808514a4: Generated new TSK 0
I20260927 00:46:50.437806 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:46:50.484767 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38767
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:50.632833 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:50.633355 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:50.633414 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:50.638106 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:50.638253 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:50.638373 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:46:50.645197 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38767
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:46:50.646931 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:50.648517 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:50.661986 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:50.661986 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:50.661980 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:50.662075 1024 server_base.cc:1034] running on GCE node
I20260927 00:46:50.662824 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:50.663901 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:50.665284 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790470010665240 us; error 101 us; skew 500 ppm
Sep 27 00:46:50 dist-test-slave-86rs krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:50.669102 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:46:50.670435 1024 webserver.cc:460] Webserver started at http://127.0.98.1:34045/ using document root <none> and password file <none>
I20260927 00:46:50.671223 1024 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:50.671331 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:50.671635 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:46:50.674556 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-0/data/instance:
uuid: "2fd6ac46cc45496fbf355d799d335130"
format_stamp: "Formatted at 2026-09-27 00:46:50 on dist-test-slave-86rs"
server_key: "b0b4cd2a51e4b2a688d6ba7360c10987"
server_key_iv: "71e7de8b02fc02168010ccb12687230a"
server_key_version: "encryptionkey@0"
I20260927 00:46:50.675273 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2fd6ac46cc45496fbf355d799d335130"
format_stamp: "Formatted at 2026-09-27 00:46:50 on dist-test-slave-86rs"
server_key: "b0b4cd2a51e4b2a688d6ba7360c10987"
server_key_iv: "71e7de8b02fc02168010ccb12687230a"
server_key_version: "encryptionkey@0"
I20260927 00:46:50.680207 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 00:46:50.683499 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:50.685719 1024 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:46:50.685932 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "2fd6ac46cc45496fbf355d799d335130"
format_stamp: "Formatted at 2026-09-27 00:46:50 on dist-test-slave-86rs"
server_key: "b0b4cd2a51e4b2a688d6ba7360c10987"
server_key_iv: "71e7de8b02fc02168010ccb12687230a"
server_key_version: "encryptionkey@0"
I20260927 00:46:50.686102 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:50.705615 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:50.710781 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:50.711014 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:50.711807 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:46:50.713059 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:46:50.713121 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:50.713184 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:46:50.713200 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:50.724957 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45713
I20260927 00:46:50.724975 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45713 every 8 connection(s)
I20260927 00:46:50.726265 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:46:50.735189 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 1024
I20260927 00:46:50.735363 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:46:50.735754 392 external_mini_cluster.cc:1428] Setting key 9a9ee7007bce988ca2fc90594aeb23ad
Sep 27 00:46:50 dist-test-slave-86rs krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:50.748057 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:50.729157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47487 (local address 127.0.98.62:38767)
0927 00:46:50.729369 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:50.729374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:50.730714 (+ 1340us) server_negotiation.cc:407] Connection header received
0927 00:46:50.732535 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:50.732539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:50.732607 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:50.732705 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:50.734560 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.735237 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.738070 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.738381 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.738521 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:50.742539 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:50.742544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:50.742546 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:50.742582 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:50.744818 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:50.745589 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:50.745593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:50.745595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:50.745705 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:50.746153 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:50.746156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:50.746157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:50.746513 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:50.746697 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:50.747625 (+ 928us) server_negotiation.cc:299] Negotiation successful
0927 00:46:50.747819 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:46:50.749308 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:50.729741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38767 (local address 127.0.98.1:47487)
0927 00:46:50.730465 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:50.730551 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:50.732083 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:50.733006 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:50.733017 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:50.733575 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:50.734310 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:50.734344 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.735726 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.735731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:50.737902 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:50.737922 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:50.738591 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:50.738597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:50.739089 (+ 492us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:50.740335 (+ 1246us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:50.740357 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:50.742266 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:50.745009 (+ 2743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:50.745026 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:50.745049 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:50.745436 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:50.745818 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:50.745822 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:50.745824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:50.746018 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:50.746693 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:50.746703 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:50.747213 (+ 510us) client_negotiation.cc:769] Sending connection context
0927 00:46:50.747505 (+ 292us) client_negotiation.cc:240] Negotiation successful
0927 00:46:50.748026 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":197,"threads_started":1}
I20260927 00:46:50.751431 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:46:50.751821 1153 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:50.752693 1153 heartbeater.cc:511] Master 127.0.98.62:38767 requested a full tablet report, sending...
I20260927 00:46:50.755455 962 ts_manager.cc:194] Registered new tserver with Master: 2fd6ac46cc45496fbf355d799d335130 (127.0.98.1:45713)
I20260927 00:46:50.757144 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:46:50.773391 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38767
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:50.916126 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:50.916584 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:50.916636 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:50.921212 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:50.921306 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:50.921379 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:46:50.927367 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38767
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:46:50.928954 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:50.930316 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:50.943356 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:50.943379 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:50.943356 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:50.943665 1158 server_base.cc:1034] running on GCE node
I20260927 00:46:50.944379 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:50.945302 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:50.946556 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790470010946510 us; error 45 us; skew 500 ppm
Sep 27 00:46:50 dist-test-slave-86rs krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:50.950361 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:46:50.951701 1158 webserver.cc:460] Webserver started at http://127.0.98.2:45277/ using document root <none> and password file <none>
I20260927 00:46:50.952628 1158 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:50.952714 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:50.953012 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:46:50.956111 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-1/data/instance:
uuid: "4a21f35f4d134579bf51a1d47958812d"
format_stamp: "Formatted at 2026-09-27 00:46:50 on dist-test-slave-86rs"
server_key: "bd1a57e19020292c4b6df7171d5d45f1"
server_key_iv: "fc087c84baa2d53934b9e553df3f4bdb"
server_key_version: "encryptionkey@0"
I20260927 00:46:50.957089 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4a21f35f4d134579bf51a1d47958812d"
format_stamp: "Formatted at 2026-09-27 00:46:50 on dist-test-slave-86rs"
server_key: "bd1a57e19020292c4b6df7171d5d45f1"
server_key_iv: "fc087c84baa2d53934b9e553df3f4bdb"
server_key_version: "encryptionkey@0"
I20260927 00:46:50.963155 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 00:46:50.966883 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:50.968809 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:46:50.968989 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "4a21f35f4d134579bf51a1d47958812d"
format_stamp: "Formatted at 2026-09-27 00:46:50 on dist-test-slave-86rs"
server_key: "bd1a57e19020292c4b6df7171d5d45f1"
server_key_iv: "fc087c84baa2d53934b9e553df3f4bdb"
server_key_version: "encryptionkey@0"
I20260927 00:46:50.969159 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:50.987036 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:50.991111 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:50.991372 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:50.992180 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:46:50.993422 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:46:50.993472 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:50.993520 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:46:50.993558 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:51.004096 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40617
I20260927 00:46:51.004261 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40617 every 8 connection(s)
I20260927 00:46:51.005344 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:46:51.013198 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 1158
I20260927 00:46:51.013346 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:46:51.013686 392 external_mini_cluster.cc:1428] Setting key 97307dcbba0a03066147dd3d37776fdb
Sep 27 00:46:51 dist-test-slave-86rs krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:51.023907 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:51.007821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36027 (local address 127.0.98.62:38767)
0927 00:46:51.008000 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:51.008006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:51.009362 (+ 1356us) server_negotiation.cc:407] Connection header received
0927 00:46:51.010955 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:51.010959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:51.011022 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:51.011112 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:51.012837 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.013687 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.015519 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.015825 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.015949 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:51.019193 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:51.019196 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:51.019199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:51.019226 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:51.020920 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:51.021699 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:51.021702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:51.021704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:51.021787 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:51.022302 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:51.022306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:51.022308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:51.022561 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:51.022654 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:51.023543 (+ 889us) server_negotiation.cc:299] Negotiation successful
0927 00:46:51.023733 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:46:51.025599 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:51.008454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38767 (local address 127.0.98.2:36027)
0927 00:46:51.009141 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:51.009206 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:51.010693 (+ 1487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:51.011391 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:51.011400 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:51.011876 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:51.012634 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:51.012654 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.013877 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.013881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:51.015385 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:51.015399 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.015998 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.016001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:51.016191 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:51.017042 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:51.017063 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:51.018892 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:51.021054 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:51.021065 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:51.021092 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:51.021560 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:51.021894 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:51.021899 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:51.021904 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:51.022157 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:51.022678 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:51.022690 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:51.023168 (+ 478us) client_negotiation.cc:769] Sending connection context
0927 00:46:51.023441 (+ 273us) client_negotiation.cc:240] Negotiation successful
0927 00:46:51.023960 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":173,"threads_started":1}
I20260927 00:46:51.027442 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38767
I20260927 00:46:51.027797 1287 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:46:51.028661 1287 heartbeater.cc:511] Master 127.0.98.62:38767 requested a full tablet report, sending...
I20260927 00:46:51.030184 962 ts_manager.cc:194] Registered new tserver with Master: 4a21f35f4d134579bf51a1d47958812d (127.0.98.2:40617)
I20260927 00:46:51.030928 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:46:51.049669 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38767
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:51.197340 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:51.197724 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:51.197778 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:51.202440 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:51.202550 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:51.202661 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:46:51.209175 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38767
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:46:51.210722 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:51.212111 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:51.224666 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:51.224656 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:51.224692 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:51.224910 1292 server_base.cc:1034] running on GCE node
I20260927 00:46:51.225575 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:51.226506 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:51.227763 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790470011227734 us; error 44 us; skew 500 ppm
Sep 27 00:46:51 dist-test-slave-86rs krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:51.231920 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:46:51.233166 1292 webserver.cc:460] Webserver started at http://127.0.98.3:35247/ using document root <none> and password file <none>
I20260927 00:46:51.233896 1292 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:51.233963 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:51.234179 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:46:51.236896 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-2/data/instance:
uuid: "4cf3067b186946f6aa1f42dafd76a136"
format_stamp: "Formatted at 2026-09-27 00:46:51 on dist-test-slave-86rs"
server_key: "7b0bbd0126ac948caf88cb2c813b4386"
server_key_iv: "5a55850f3328c353b67071654906a93e"
server_key_version: "encryptionkey@0"
I20260927 00:46:51.237512 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4cf3067b186946f6aa1f42dafd76a136"
format_stamp: "Formatted at 2026-09-27 00:46:51 on dist-test-slave-86rs"
server_key: "7b0bbd0126ac948caf88cb2c813b4386"
server_key_iv: "5a55850f3328c353b67071654906a93e"
server_key_version: "encryptionkey@0"
I20260927 00:46:51.242611 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 00:46:51.246140 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:51.247979 1292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:46:51.248174 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "4cf3067b186946f6aa1f42dafd76a136"
format_stamp: "Formatted at 2026-09-27 00:46:51 on dist-test-slave-86rs"
server_key: "7b0bbd0126ac948caf88cb2c813b4386"
server_key_iv: "5a55850f3328c353b67071654906a93e"
server_key_version: "encryptionkey@0"
I20260927 00:46:51.248397 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:51.266755 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:51.271059 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:51.271323 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:51.272181 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:46:51.273324 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:46:51.273403 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:51.273485 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:46:51.273537 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:51.284446 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34293
I20260927 00:46:51.284473 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34293 every 8 connection(s)
I20260927 00:46:51.286089 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:46:51.288767 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 1292
I20260927 00:46:51.288906 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790470006085796-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:46:51.289259 392 external_mini_cluster.cc:1428] Setting key 5121972b0c86bea685a2e106ab1169ac
Sep 27 00:46:51 dist-test-slave-86rs krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:51.303998 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:51.288551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40873 (local address 127.0.98.62:38767)
0927 00:46:51.288728 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:51.288732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:51.290105 (+ 1373us) server_negotiation.cc:407] Connection header received
0927 00:46:51.291972 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:51.291975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:51.292048 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:51.292135 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:51.293754 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.294361 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.295669 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.295905 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.296010 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:51.299260 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:51.299264 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:51.299267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:51.299301 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:51.301115 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:51.301785 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:51.301789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:51.301790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:51.301887 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:51.302369 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:51.302373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:51.302374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:51.302656 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:51.302786 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:51.303649 (+ 863us) server_negotiation.cc:299] Negotiation successful
0927 00:46:51.303813 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:46:51.305393 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:51.289253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38767 (local address 127.0.98.3:40873)
0927 00:46:51.289932 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:51.289994 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:51.291665 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:51.292416 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:51.292426 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:51.292928 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:51.293567 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:51.293608 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.294483 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.294486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:51.295531 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:51.295540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.296060 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.296063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:51.296311 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:51.297065 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:51.297085 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:51.299000 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:51.301273 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:51.301280 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:51.301302 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:51.301661 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:51.302004 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:51.302007 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:51.302009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:51.302226 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:51.302785 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:51.302796 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:51.303255 (+ 459us) client_negotiation.cc:769] Sending connection context
0927 00:46:51.303601 (+ 346us) client_negotiation.cc:240] Negotiation successful
0927 00:46:51.304077 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":176,"threads_started":1}
I20260927 00:46:51.306911 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38767
I20260927 00:46:51.307271 1421 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:51.307914 1421 heartbeater.cc:511] Master 127.0.98.62:38767 requested a full tablet report, sending...
I20260927 00:46:51.309274 962 ts_manager.cc:194] Registered new tserver with Master: 4cf3067b186946f6aa1f42dafd76a136 (127.0.98.3:34293)
I20260927 00:46:51.309996 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40873
I20260927 00:46:51.317094 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:46:51.330983 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:51.321924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57266 (local address 127.0.98.62:38767)
0927 00:46:51.322152 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:51.322157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:51.322253 (+ 96us) server_negotiation.cc:407] Connection header received
0927 00:46:51.322397 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:51.322400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:51.322486 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:51.322586 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:51.323864 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.324549 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.325462 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.325703 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.325790 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:51.327435 (+ 1645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:51.327439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:51.327441 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:51.327469 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:51.329253 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:51.329670 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:51.329673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:51.329674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:51.329753 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:51.329985 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:51.329987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:51.329988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:51.330313 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:51.330429 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:51.330643 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 00:46:51.330755 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260927 00:46:51.338313 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57266:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:46:51.342712 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:46:51.362414 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:51.355441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40617 (local address 127.0.0.1:48514)
0927 00:46:51.356361 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:51.356437 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:51.356585 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:51.357470 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:51.357477 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:51.357502 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:51.357701 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:51.357708 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.359112 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.359123 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:51.360499 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:51.360509 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.361657 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.361661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:51.361792 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:51.361846 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 00:46:51.362023 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 00:46:51.362201 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":1,"thread_start_us":134,"threads_started":1}
I20260927 00:46:51.362622 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:51.355530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34293 (local address 127.0.0.1:39380)
0927 00:46:51.357038 (+ 1508us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:51.357055 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:51.357169 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:51.358303 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:51.358305 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:51.358323 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:51.358420 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:51.358425 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.359842 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.359846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:51.361024 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:51.361033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.362127 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.362131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:51.362259 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:51.362283 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:46:51.362346 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 00:46:51.362419 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1445,"mutex_wait_us":18,"thread_start_us":165,"threads_started":1}
I20260927 00:46:51.362776 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:51.356531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39380 (local address 127.0.98.3:34293)
0927 00:46:51.357786 (+ 1255us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:51.357801 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:51.357826 (+ 25us) server_negotiation.cc:407] Connection header received
0927 00:46:51.357916 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:51.357924 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:51.358099 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:51.358321 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:51.358532 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.359680 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.361185 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.361978 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.362231 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:51.362396 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:51.362453 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 00:46:51.362577 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1141,"thread_start_us":200,"threads_started":1}
I20260927 00:46:51.363359 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:51.355646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48514 (local address 127.0.98.2:40617)
0927 00:46:51.356945 (+ 1299us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:51.356959 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:51.356985 (+ 26us) server_negotiation.cc:407] Connection header received
0927 00:46:51.357082 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:51.357087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:51.357292 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:51.357488 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:51.357816 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.358917 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.360635 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.361502 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.361636 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:51.361836 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:51.362068 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 00:46:51.362203 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1170,"thread_start_us":174,"threads_started":1}
I20260927 00:46:51.363457 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:51.355861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45713 (local address 127.0.0.1:47774)
0927 00:46:51.357327 (+ 1466us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:51.357338 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:51.357443 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:51.357990 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:51.357993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:51.358010 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:51.358114 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:51.358121 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.359631 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.359634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:51.360946 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:51.360958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.362143 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.362147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:51.363043 (+ 896us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:51.363065 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:46:51.363120 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 00:46:51.363194 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1418,"mutex_wait_us":1,"thread_start_us":140,"threads_started":1}
I20260927 00:46:51.364146 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:51.356562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47774 (local address 127.0.98.1:45713)
0927 00:46:51.357056 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:51.357066 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:51.357427 (+ 361us) server_negotiation.cc:407] Connection header received
0927 00:46:51.357607 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:51.357615 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:51.357839 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:51.358015 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:51.358244 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.359465 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.361140 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.361963 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.363638 (+ 1675us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:51.363799 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:51.363857 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 00:46:51.363990 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":88,"threads_started":1}
I20260927 00:46:51.366932 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 74b59559b633462fa52e7f44ad48230d (DEFAULT_TABLE table=test-table [id=b0f7bcf637ff4db7b612d55d60718080]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:51.366932 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 74b59559b633462fa52e7f44ad48230d (DEFAULT_TABLE table=test-table [id=b0f7bcf637ff4db7b612d55d60718080]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:51.366853 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 74b59559b633462fa52e7f44ad48230d (DEFAULT_TABLE table=test-table [id=b0f7bcf637ff4db7b612d55d60718080]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:51.368498 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74b59559b633462fa52e7f44ad48230d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:51.368510 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74b59559b633462fa52e7f44ad48230d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:51.368501 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74b59559b633462fa52e7f44ad48230d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:51.380605 1439 tablet_bootstrap.cc:492] T 74b59559b633462fa52e7f44ad48230d P 4a21f35f4d134579bf51a1d47958812d: Bootstrap starting.
I20260927 00:46:51.381319 1438 tablet_bootstrap.cc:492] T 74b59559b633462fa52e7f44ad48230d P 4cf3067b186946f6aa1f42dafd76a136: Bootstrap starting.
I20260927 00:46:51.383796 1438 tablet_bootstrap.cc:654] T 74b59559b633462fa52e7f44ad48230d P 4cf3067b186946f6aa1f42dafd76a136: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:51.383862 1439 tablet_bootstrap.cc:654] T 74b59559b633462fa52e7f44ad48230d P 4a21f35f4d134579bf51a1d47958812d: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:51.383818 1440 tablet_bootstrap.cc:492] T 74b59559b633462fa52e7f44ad48230d P 2fd6ac46cc45496fbf355d799d335130: Bootstrap starting.
I20260927 00:46:51.385470 1438 log.cc:824] T 74b59559b633462fa52e7f44ad48230d P 4cf3067b186946f6aa1f42dafd76a136: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:51.385473 1439 log.cc:824] T 74b59559b633462fa52e7f44ad48230d P 4a21f35f4d134579bf51a1d47958812d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:51.387105 1440 tablet_bootstrap.cc:654] T 74b59559b633462fa52e7f44ad48230d P 2fd6ac46cc45496fbf355d799d335130: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:51.388162 1440 log.cc:824] T 74b59559b633462fa52e7f44ad48230d P 2fd6ac46cc45496fbf355d799d335130: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:51.388420 1439 tablet_bootstrap.cc:492] T 74b59559b633462fa52e7f44ad48230d P 4a21f35f4d134579bf51a1d47958812d: No bootstrap required, opened a new log
I20260927 00:46:51.388420 1438 tablet_bootstrap.cc:492] T 74b59559b633462fa52e7f44ad48230d P 4cf3067b186946f6aa1f42dafd76a136: No bootstrap required, opened a new log
I20260927 00:46:51.388666 1438 ts_tablet_manager.cc:1397] T 74b59559b633462fa52e7f44ad48230d P 4cf3067b186946f6aa1f42dafd76a136: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260927 00:46:51.388674 1439 ts_tablet_manager.cc:1397] T 74b59559b633462fa52e7f44ad48230d P 4a21f35f4d134579bf51a1d47958812d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20260927 00:46:51.391067 1440 tablet_bootstrap.cc:492] T 74b59559b633462fa52e7f44ad48230d P 2fd6ac46cc45496fbf355d799d335130: No bootstrap required, opened a new log
I20260927 00:46:51.391372 1440 ts_tablet_manager.cc:1397] T 74b59559b633462fa52e7f44ad48230d P 2fd6ac46cc45496fbf355d799d335130: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260927 00:46:51.392896 1438 raft_consensus.cc:348] T 74b59559b633462fa52e7f44ad48230d P 4cf3067b186946f6aa1f42dafd76a136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a21f35f4d134579bf51a1d47958812d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40617 } } peers { permanent_uuid: "4cf3067b186946f6aa1f42dafd76a136" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34293 } } peers { permanent_uuid: "2fd6ac46cc45496fbf355d799d335130" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45713 } }
I20260927 00:46:51.393327 1438 raft_consensus.cc:374] T 74b59559b633462fa52e7f44ad48230d P 4cf3067b186946f6aa1f42dafd76a136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:51.393404 1438 raft_consensus.cc:729] T 74b59559b633462fa52e7f44ad48230d P 4cf3067b186946f6aa1f42dafd76a136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cf3067b186946f6aa1f42dafd76a136, State: Initialized, Role: FOLLOWER
I20260927 00:46:51.393261 1439 raft_consensus.cc:348] T 74b59559b633462fa52e7f44ad48230d P 4a21f35f4d134579bf51a1d47958812d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a21f35f4d134579bf51a1d47958812d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40617 } } peers { permanent_uuid: "4cf3067b186946f6aa1f42dafd76a136" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34293 } } peers { permanent_uuid: "2fd6ac46cc45496fbf355d799d335130" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45713 } }
I20260927 00:46:51.393699 1439 raft_consensus.cc:374] T 74b59559b633462fa52e7f44ad48230d P 4a21f35f4d134579bf51a1d47958812d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:51.393787 1439 raft_consensus.cc:729] T 74b59559b633462fa52e7f44ad48230d P 4a21f35f4d134579bf51a1d47958812d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a21f35f4d134579bf51a1d47958812d, State: Initialized, Role: FOLLOWER
I20260927 00:46:51.394177 1438 consensus_queue.cc:260] T 74b59559b633462fa52e7f44ad48230d P 4cf3067b186946f6aa1f42dafd76a136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a21f35f4d134579bf51a1d47958812d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40617 } } peers { permanent_uuid: "4cf3067b186946f6aa1f42dafd76a136" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34293 } } peers { permanent_uuid: "2fd6ac46cc45496fbf355d799d335130" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45713 } }
I20260927 00:46:51.394356 1439 consensus_queue.cc:260] T 74b59559b633462fa52e7f44ad48230d P 4a21f35f4d134579bf51a1d47958812d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a21f35f4d134579bf51a1d47958812d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40617 } } peers { permanent_uuid: "4cf3067b186946f6aa1f42dafd76a136" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34293 } } peers { permanent_uuid: "2fd6ac46cc45496fbf355d799d335130" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45713 } }
I20260927 00:46:51.395114 1421 heartbeater.cc:503] Master 127.0.98.62:38767 was elected leader, sending a full tablet report...
I20260927 00:46:51.395615 1438 ts_tablet_manager.cc:1428] T 74b59559b633462fa52e7f44ad48230d P 4cf3067b186946f6aa1f42dafd76a136: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 00:46:51.395617 1439 ts_tablet_manager.cc:1428] T 74b59559b633462fa52e7f44ad48230d P 4a21f35f4d134579bf51a1d47958812d: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260927 00:46:51.396049 1287 heartbeater.cc:503] Master 127.0.98.62:38767 was elected leader, sending a full tablet report...
I20260927 00:46:51.396963 1440 raft_consensus.cc:348] T 74b59559b633462fa52e7f44ad48230d P 2fd6ac46cc45496fbf355d799d335130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a21f35f4d134579bf51a1d47958812d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40617 } } peers { permanent_uuid: "4cf3067b186946f6aa1f42dafd76a136" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34293 } } peers { permanent_uuid: "2fd6ac46cc45496fbf355d799d335130" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45713 } }
I20260927 00:46:51.397310 1440 raft_consensus.cc:374] T 74b59559b633462fa52e7f44ad48230d P 2fd6ac46cc45496fbf355d799d335130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:51.397413 1440 raft_consensus.cc:729] T 74b59559b633462fa52e7f44ad48230d P 2fd6ac46cc45496fbf355d799d335130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fd6ac46cc45496fbf355d799d335130, State: Initialized, Role: FOLLOWER
I20260927 00:46:51.397965 1440 consensus_queue.cc:260] T 74b59559b633462fa52e7f44ad48230d P 2fd6ac46cc45496fbf355d799d335130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a21f35f4d134579bf51a1d47958812d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40617 } } peers { permanent_uuid: "4cf3067b186946f6aa1f42dafd76a136" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34293 } } peers { permanent_uuid: "2fd6ac46cc45496fbf355d799d335130" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45713 } }
I20260927 00:46:51.398773 1153 heartbeater.cc:503] Master 127.0.98.62:38767 was elected leader, sending a full tablet report...
I20260927 00:46:51.399276 1440 ts_tablet_manager.cc:1428] T 74b59559b633462fa52e7f44ad48230d P 2fd6ac46cc45496fbf355d799d335130: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
W20260927 00:46:51.478319 1154 tablet.cc:2369] T 74b59559b633462fa52e7f44ad48230d P 2fd6ac46cc45496fbf355d799d335130: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:46:51.506829 1288 tablet.cc:2369] T 74b59559b633462fa52e7f44ad48230d P 4a21f35f4d134579bf51a1d47958812d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:46:51.537384 1422 tablet.cc:2369] T 74b59559b633462fa52e7f44ad48230d P 4cf3067b186946f6aa1f42dafd76a136: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:46:51.616951 1444 raft_consensus.cc:482] T 74b59559b633462fa52e7f44ad48230d P 4cf3067b186946f6aa1f42dafd76a136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:46:51.617269 1444 raft_consensus.cc:504] T 74b59559b633462fa52e7f44ad48230d P 4cf3067b186946f6aa1f42dafd76a136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a21f35f4d134579bf51a1d47958812d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40617 } } peers { permanent_uuid: "4cf3067b186946f6aa1f42dafd76a136" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34293 } } peers { permanent_uuid: "2fd6ac46cc45496fbf355d799d335130" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45713 } }
I20260927 00:46:51.619073 1444 leader_election.cc:288] T 74b59559b633462fa52e7f44ad48230d P 4cf3067b186946f6aa1f42dafd76a136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a21f35f4d134579bf51a1d47958812d (127.0.98.2:40617), 2fd6ac46cc45496fbf355d799d335130 (127.0.98.1:45713)
I20260927 00:46:51.625741 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:51.619515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36407 (local address 127.0.98.2:40617)
0927 00:46:51.619677 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:51.619684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:51.620013 (+ 329us) server_negotiation.cc:407] Connection header received
0927 00:46:51.620131 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:51.620136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:51.620192 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:51.620329 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:51.620631 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.621369 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.623920 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.624636 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.624768 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:51.624856 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:51.625437 (+ 581us) server_negotiation.cc:299] Negotiation successful
0927 00:46:51.625542 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 00:46:51.625722 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:51.619712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56657 (local address 127.0.98.1:45713)
0927 00:46:51.619875 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:51.619879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:51.619895 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:46:51.619954 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:51.619956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:51.620004 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:51.620110 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:51.620422 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.621281 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.623920 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.624650 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.624768 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:51.624856 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:51.625437 (+ 581us) server_negotiation.cc:299] Negotiation successful
0927 00:46:51.625542 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 00:46:51.625870 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:51.619382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45713 (local address 127.0.98.3:56657)
0927 00:46:51.619628 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:51.619646 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:51.619772 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:51.620124 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:51.620127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:51.620145 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:51.620307 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:51.620314 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.621412 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.621448 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:51.623729 (+ 2281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:51.623744 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.624834 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.624838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:51.625183 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:51.625286 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 00:46:51.625396 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 00:46:51.625650 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":271}
I20260927 00:46:51.626149 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:51.619391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40617 (local address 127.0.98.3:36407)
0927 00:46:51.619910 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:51.619920 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:51.620042 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:51.620337 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:51.620340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:51.620355 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:51.620481 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:51.620487 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.621487 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.621489 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:51.623729 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:51.623744 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.624834 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.624838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:51.625185 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:51.625286 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 00:46:51.625396 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 00:46:51.625650 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":287,"thread_start_us":193,"threads_started":1}
I20260927 00:46:51.626822 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b59559b633462fa52e7f44ad48230d" candidate_uuid: "4cf3067b186946f6aa1f42dafd76a136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fd6ac46cc45496fbf355d799d335130" is_pre_election: true
I20260927 00:46:51.626960 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b59559b633462fa52e7f44ad48230d" candidate_uuid: "4cf3067b186946f6aa1f42dafd76a136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a21f35f4d134579bf51a1d47958812d" is_pre_election: true
I20260927 00:46:51.627288 1107 raft_consensus.cc:2454] T 74b59559b633462fa52e7f44ad48230d P 2fd6ac46cc45496fbf355d799d335130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cf3067b186946f6aa1f42dafd76a136 in term 0.
I20260927 00:46:51.627375 1241 raft_consensus.cc:2454] T 74b59559b633462fa52e7f44ad48230d P 4a21f35f4d134579bf51a1d47958812d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cf3067b186946f6aa1f42dafd76a136 in term 0.
I20260927 00:46:51.627964 1311 leader_election.cc:302] T 74b59559b633462fa52e7f44ad48230d P 4cf3067b186946f6aa1f42dafd76a136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2fd6ac46cc45496fbf355d799d335130, 4a21f35f4d134579bf51a1d47958812d, 4cf3067b186946f6aa1f42dafd76a136; no voters:
I20260927 00:46:51.628379 1444 raft_consensus.cc:2781] T 74b59559b633462fa52e7f44ad48230d P 4cf3067b186946f6aa1f42dafd76a136 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:46:51.628486 1444 raft_consensus.cc:482] T 74b59559b633462fa52e7f44ad48230d P 4cf3067b186946f6aa1f42dafd76a136 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:46:51.628530 1444 raft_consensus.cc:3037] T 74b59559b633462fa52e7f44ad48230d P 4cf3067b186946f6aa1f42dafd76a136 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:51.629835 1444 raft_consensus.cc:504] T 74b59559b633462fa52e7f44ad48230d P 4cf3067b186946f6aa1f42dafd76a136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a21f35f4d134579bf51a1d47958812d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40617 } } peers { permanent_uuid: "4cf3067b186946f6aa1f42dafd76a136" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34293 } } peers { permanent_uuid: "2fd6ac46cc45496fbf355d799d335130" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45713 } }
I20260927 00:46:51.630316 1444 leader_election.cc:288] T 74b59559b633462fa52e7f44ad48230d P 4cf3067b186946f6aa1f42dafd76a136 [CANDIDATE]: Term 1 election: Requested vote from peers 4a21f35f4d134579bf51a1d47958812d (127.0.98.2:40617), 2fd6ac46cc45496fbf355d799d335130 (127.0.98.1:45713)
I20260927 00:46:51.630738 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b59559b633462fa52e7f44ad48230d" candidate_uuid: "4cf3067b186946f6aa1f42dafd76a136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fd6ac46cc45496fbf355d799d335130"
I20260927 00:46:51.630782 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b59559b633462fa52e7f44ad48230d" candidate_uuid: "4cf3067b186946f6aa1f42dafd76a136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a21f35f4d134579bf51a1d47958812d"
I20260927 00:46:51.630896 1107 raft_consensus.cc:3037] T 74b59559b633462fa52e7f44ad48230d P 2fd6ac46cc45496fbf355d799d335130 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:51.630963 1241 raft_consensus.cc:3037] T 74b59559b633462fa52e7f44ad48230d P 4a21f35f4d134579bf51a1d47958812d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:51.632362 1107 raft_consensus.cc:2454] T 74b59559b633462fa52e7f44ad48230d P 2fd6ac46cc45496fbf355d799d335130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cf3067b186946f6aa1f42dafd76a136 in term 1.
I20260927 00:46:51.632494 1241 raft_consensus.cc:2454] T 74b59559b633462fa52e7f44ad48230d P 4a21f35f4d134579bf51a1d47958812d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cf3067b186946f6aa1f42dafd76a136 in term 1.
I20260927 00:46:51.632860 1311 leader_election.cc:302] T 74b59559b633462fa52e7f44ad48230d P 4cf3067b186946f6aa1f42dafd76a136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fd6ac46cc45496fbf355d799d335130, 4cf3067b186946f6aa1f42dafd76a136; no voters:
I20260927 00:46:51.633100 1444 raft_consensus.cc:2781] T 74b59559b633462fa52e7f44ad48230d P 4cf3067b186946f6aa1f42dafd76a136 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:46:51.633392 1444 raft_consensus.cc:686] T 74b59559b633462fa52e7f44ad48230d P 4cf3067b186946f6aa1f42dafd76a136 [term 1 LEADER]: Becoming Leader. State: Replica: 4cf3067b186946f6aa1f42dafd76a136, State: Running, Role: LEADER
I20260927 00:46:51.633982 1444 consensus_queue.cc:237] T 74b59559b633462fa52e7f44ad48230d P 4cf3067b186946f6aa1f42dafd76a136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a21f35f4d134579bf51a1d47958812d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40617 } } peers { permanent_uuid: "4cf3067b186946f6aa1f42dafd76a136" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34293 } } peers { permanent_uuid: "2fd6ac46cc45496fbf355d799d335130" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45713 } }
I20260927 00:46:51.638940 961 catalog_manager.cc:5495] T 74b59559b633462fa52e7f44ad48230d P 4cf3067b186946f6aa1f42dafd76a136 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cf3067b186946f6aa1f42dafd76a136 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4cf3067b186946f6aa1f42dafd76a136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a21f35f4d134579bf51a1d47958812d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cf3067b186946f6aa1f42dafd76a136" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fd6ac46cc45496fbf355d799d335130" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45713 } health_report { overall_health: UNKNOWN } } }
Sep 27 00:46:51 dist-test-slave-86rs krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:46:51.673863 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:51.661935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47780 (local address 127.0.98.1:45713)
0927 00:46:51.662172 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:51.662176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:51.662290 (+ 114us) server_negotiation.cc:407] Connection header received
0927 00:46:51.662507 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:51.662510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:51.662568 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:51.662659 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:51.663642 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.664274 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.665204 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.665437 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.665529 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:51.668909 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:51.668928 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:51.668950 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:51.668991 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:51.671349 (+ 2358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:51.671984 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:51.671991 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:51.671993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:51.672137 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:51.672535 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:51.672537 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:51.672539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:51.672873 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:51.673071 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:51.673372 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 00:46:51.673550 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
Sep 27 00:46:51 dist-test-slave-86rs krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470011, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:46:51 dist-test-slave-86rs krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470011, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:46:51.708287 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:51.697264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47784 (local address 127.0.98.1:45713)
0927 00:46:51.697514 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:51.697520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:51.697694 (+ 174us) server_negotiation.cc:407] Connection header received
0927 00:46:51.697929 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:51.697933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:51.697986 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:51.698084 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:51.698896 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.699496 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.700483 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.700731 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.700848 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:51.704199 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:51.704204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:51.704206 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:51.704301 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:51.706133 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:51.706725 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:51.706729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:51.706731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:51.706835 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:51.707150 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:51.707155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:51.707157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:51.707382 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:51.707537 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:51.707879 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 00:46:51.708037 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
Sep 27 00:46:51 dist-test-slave-86rs krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:46:51 dist-test-slave-86rs krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:46:51.739068 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:51.728989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47794 (local address 127.0.98.1:45713)
0927 00:46:51.729166 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:51.729172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:51.729389 (+ 217us) server_negotiation.cc:407] Connection header received
0927 00:46:51.729578 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:51.729582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:51.729641 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:51.729736 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:51.730544 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.731123 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.732010 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:51.732297 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:51.732408 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:51.735173 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:51.735176 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:51.735179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:51.735204 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:51.736961 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:51.737490 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:51.737494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:51.737496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:51.737586 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:51.737941 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:51.737947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:51.737949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:51.738170 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:51.738332 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:51.738609 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 00:46:51.738770 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 00:46:51.740957 1067 tablet_service.cc:2908] Rejecting scan request for tablet 74b59559b633462fa52e7f44ad48230d: Uninitialized: clean time has not yet been initialized
I20260927 00:46:51.743938 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 1024
I20260927 00:46:51.751820 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 1158
I20260927 00:46:51.759459 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 1292
I20260927 00:46:51.766340 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 930
2026-09-27T00:46:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3161 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:46:51 dist-test-slave-86rs krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:46:51 dist-test-slave-86rs krb5kdc[1462](info): set up 2 sockets
Sep 27 00:46:51 dist-test-slave-86rs krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260927 00:46:52.832593 392 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:46:52 dist-test-slave-86rs krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:46:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:46:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:46:52.980762 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44639
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:53.126391 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:53.126842 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:53.126893 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:53.131489 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:46:53.131585 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 00:46:53.131605 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:53.131625 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:53.131649 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:46:53.131664 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:46:53.137974 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44639
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:46:53.139469 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:53.140898 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:46:53.152275 1475 server_base.cc:1034] running on GCE node
W20260927 00:46:53.152149 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:53.152175 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:53.152163 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:53.153564 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:53.155555 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:53.156863 1475 hybrid_clock.cc:648] HybridClock initialized: now 1790470013156819 us; error 73 us; skew 500 ppm
Sep 27 00:46:53 dist-test-slave-86rs krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:53.160970 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:46:53.162508 1475 webserver.cc:460] Webserver started at http://127.0.98.62:44615/ using document root <none> and password file <none>
I20260927 00:46:53.163350 1475 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:53.163444 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:53.163712 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:46:53.166926 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "3022093688014114898d44f408dda928"
format_stamp: "Formatted at 2026-09-27 00:46:53 on dist-test-slave-86rs"
server_key: "dfd5bce9f7c9485e8a514f79524e4502"
server_key_iv: "5c107153a43712e7034c448e735a4958"
server_key_version: "encryptionkey@0"
I20260927 00:46:53.167786 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "3022093688014114898d44f408dda928"
format_stamp: "Formatted at 2026-09-27 00:46:53 on dist-test-slave-86rs"
server_key: "dfd5bce9f7c9485e8a514f79524e4502"
server_key_iv: "5c107153a43712e7034c448e735a4958"
server_key_version: "encryptionkey@0"
I20260927 00:46:53.175138 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.001s
I20260927 00:46:53.178974 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:53.181022 1475 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:46:53.181288 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "3022093688014114898d44f408dda928"
format_stamp: "Formatted at 2026-09-27 00:46:53 on dist-test-slave-86rs"
server_key: "dfd5bce9f7c9485e8a514f79524e4502"
server_key_iv: "5c107153a43712e7034c448e735a4958"
server_key_version: "encryptionkey@0"
I20260927 00:46:53.181475 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:53.204047 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:53.208609 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:53.208896 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:53.219419 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44639
I20260927 00:46:53.219434 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44639 every 8 connection(s)
I20260927 00:46:53.221095 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:46:53.225337 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:53.230909 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 1475
I20260927 00:46:53.231069 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:46:53.231379 392 external_mini_cluster.cc:1428] Setting key f5ff96c3dde36274a07b655378646f28
I20260927 00:46:53.233474 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3022093688014114898d44f408dda928: Bootstrap starting.
I20260927 00:46:53.236825 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3022093688014114898d44f408dda928: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:53.238253 1553 log.cc:824] T 00000000000000000000000000000000 P 3022093688014114898d44f408dda928: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:53.241791 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3022093688014114898d44f408dda928: No bootstrap required, opened a new log
I20260927 00:46:53.246368 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3022093688014114898d44f408dda928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3022093688014114898d44f408dda928" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44639 } }
I20260927 00:46:53.246788 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3022093688014114898d44f408dda928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:53.246856 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3022093688014114898d44f408dda928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3022093688014114898d44f408dda928, State: Initialized, Role: FOLLOWER
I20260927 00:46:53.247484 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3022093688014114898d44f408dda928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3022093688014114898d44f408dda928" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44639 } }
I20260927 00:46:53.247687 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3022093688014114898d44f408dda928 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:46:53.247786 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3022093688014114898d44f408dda928 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:46:53.247920 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3022093688014114898d44f408dda928 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 00:46:53 dist-test-slave-86rs krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:53.249434 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3022093688014114898d44f408dda928 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3022093688014114898d44f408dda928" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44639 } }
I20260927 00:46:53.250057 1553 leader_election.cc:302] T 00000000000000000000000000000000 P 3022093688014114898d44f408dda928 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3022093688014114898d44f408dda928; no voters:
I20260927 00:46:53.250633 1553 leader_election.cc:288] T 00000000000000000000000000000000 P 3022093688014114898d44f408dda928 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:46:53.250835 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3022093688014114898d44f408dda928 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:46:53.251516 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3022093688014114898d44f408dda928 [term 1 LEADER]: Becoming Leader. State: Replica: 3022093688014114898d44f408dda928, State: Running, Role: LEADER
I20260927 00:46:53.252097 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3022093688014114898d44f408dda928 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3022093688014114898d44f408dda928" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44639 } }
I20260927 00:46:53.252748 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3022093688014114898d44f408dda928 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:46:53.254604 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3022093688014114898d44f408dda928 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3022093688014114898d44f408dda928" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3022093688014114898d44f408dda928" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44639 } } }
I20260927 00:46:53.254777 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3022093688014114898d44f408dda928 [sys.catalog]: This master's current role is: LEADER
I20260927 00:46:53.257390 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3022093688014114898d44f408dda928 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3022093688014114898d44f408dda928. Latest consensus state: current_term: 1 leader_uuid: "3022093688014114898d44f408dda928" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3022093688014114898d44f408dda928" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44639 } } }
I20260927 00:46:53.257551 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3022093688014114898d44f408dda928 [sys.catalog]: This master's current role is: LEADER
I20260927 00:46:53.259022 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.233450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49092 (local address 127.0.98.62:44639)
0927 00:46:53.234035 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:53.234096 (+ 61us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:53.234147 (+ 51us) server_negotiation.cc:407] Connection header received
0927 00:46:53.235205 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:53.235236 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:53.235805 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:53.236369 (+ 564us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:53.237430 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.239348 (+ 1918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.240952 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.241855 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.242098 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.250499 (+ 8401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:53.250511 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:53.250546 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:53.250615 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:53.253181 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.253983 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.253997 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.254012 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.254197 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.255603 (+ 1406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.255607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.255609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.256302 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:53.256568 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:53.257460 (+ 892us) server_negotiation.cc:299] Negotiation successful
0927 00:46:53.257954 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":177,"threads_started":1}
W20260927 00:46:53.259511 1573 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3022093688014114898d44f408dda928: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:46:53.259672 1573 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:46:53.259775 1571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:46:53.260919 1571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:46:53.268673 1571 catalog_manager.cc:1347] Generated new cluster ID: 5d7614fa46c04d2fbc0b1a7c7d1a4edd
I20260927 00:46:53.268784 1571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:46:53.311635 1571 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:46:53.313192 1571 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:46:53.329021 1571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3022093688014114898d44f408dda928: Generated new TSK 0
I20260927 00:46:53.330224 1571 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:46:53.372217 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44639
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:53.537976 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:53.538421 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:53.538512 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:53.542994 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:53.543105 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:53.543187 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:46:53.543282 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:46:53.549645 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:46:53.551281 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:53.552914 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:53.566179 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:53.566179 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:53.566179 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:53.566319 1579 server_base.cc:1034] running on GCE node
I20260927 00:46:53.567099 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:53.568028 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:53.569278 1579 hybrid_clock.cc:648] HybridClock initialized: now 1790470013569249 us; error 46 us; skew 500 ppm
Sep 27 00:46:53 dist-test-slave-86rs krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:53.572881 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:46:53.573999 1579 webserver.cc:460] Webserver started at http://127.0.98.1:46075/ using document root <none> and password file <none>
I20260927 00:46:53.574757 1579 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:53.574832 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:53.575050 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:46:53.577780 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-0/data/instance:
uuid: "9b56cc27420a49a3b5569894cbbe9b6e"
format_stamp: "Formatted at 2026-09-27 00:46:53 on dist-test-slave-86rs"
server_key: "6abeaaff48750791d788f8620b6639ca"
server_key_iv: "188f7a4f84b413738ce66020a4c7d5af"
server_key_version: "encryptionkey@0"
I20260927 00:46:53.578423 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9b56cc27420a49a3b5569894cbbe9b6e"
format_stamp: "Formatted at 2026-09-27 00:46:53 on dist-test-slave-86rs"
server_key: "6abeaaff48750791d788f8620b6639ca"
server_key_iv: "188f7a4f84b413738ce66020a4c7d5af"
server_key_version: "encryptionkey@0"
I20260927 00:46:53.583461 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 00:46:53.586843 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:53.588968 1579 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 00:46:53.589149 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "9b56cc27420a49a3b5569894cbbe9b6e"
format_stamp: "Formatted at 2026-09-27 00:46:53 on dist-test-slave-86rs"
server_key: "6abeaaff48750791d788f8620b6639ca"
server_key_iv: "188f7a4f84b413738ce66020a4c7d5af"
server_key_version: "encryptionkey@0"
I20260927 00:46:53.589300 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:53.610890 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:53.615794 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:53.616047 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:53.618076 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:46:53.618149 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:53.618202 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:46:53.618261 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:53.631254 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40385
I20260927 00:46:53.631305 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40385 every 8 connection(s)
Sep 27 00:46:53 dist-test-slave-86rs krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:53.632855 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:46:53.633077 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 1579
I20260927 00:46:53.633208 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:46:53.633610 392 external_mini_cluster.cc:1428] Setting key 409480d5625f2dbbfda2d248214c13e0
I20260927 00:46:53.637984 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.619461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35027 (local address 127.0.98.62:44639)
0927 00:46:53.619674 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:53.619681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:53.621054 (+ 1373us) server_negotiation.cc:407] Connection header received
0927 00:46:53.622970 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:53.622974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:53.623040 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:53.623127 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:53.625734 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.626776 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.628443 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.628817 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.628960 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.632452 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:53.632456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:53.632459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:53.632496 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:53.634747 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.635647 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.635652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.635654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.635785 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.636205 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.636210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.636212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.636588 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:53.636774 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:53.637486 (+ 712us) server_negotiation.cc:299] Negotiation successful
0927 00:46:53.637698 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 00:46:53.639492 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.620062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44639 (local address 127.0.98.1:35027)
0927 00:46:53.620808 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:53.620900 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:53.622619 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:53.623464 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:53.623481 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:53.624263 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:53.625409 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:53.625453 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.626949 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.626954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:53.628215 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:53.628290 (+ 75us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.629022 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.629027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:53.629257 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.630040 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:53.630061 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:53.632146 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:53.634916 (+ 2770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:53.634925 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:53.634949 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:53.635439 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:53.635897 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:53.635900 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:53.635902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:53.636082 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:53.636785 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:53.636792 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:53.637152 (+ 360us) client_negotiation.cc:769] Sending connection context
0927 00:46:53.637370 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 00:46:53.637823 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":212,"threads_started":1}
I20260927 00:46:53.641645 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44639
I20260927 00:46:53.641896 1710 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:53.642911 1710 heartbeater.cc:511] Master 127.0.98.62:44639 requested a full tablet report, sending...
I20260927 00:46:53.645583 1504 ts_manager.cc:194] Registered new tserver with Master: 9b56cc27420a49a3b5569894cbbe9b6e (127.0.98.1:40385)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:46:53.647487 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:46:53.658291 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.647475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53915 (local address 127.0.98.62:44639)
0927 00:46:53.647666 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:53.647672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:53.647697 (+ 25us) server_negotiation.cc:407] Connection header received
0927 00:46:53.647776 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:53.647779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:53.647830 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:53.647961 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:53.648964 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.649579 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.650877 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.651246 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.651423 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.653245 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:53.653251 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:53.653254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:53.653287 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:53.655607 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.656218 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.656261 (+ 43us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.656263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.656371 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.656711 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.656715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.656818 (+ 103us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.657097 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:53.657308 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:53.657699 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 00:46:53.657869 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 00:46:53.658296 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.644873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44639 (local address 127.0.98.1:53915)
0927 00:46:53.645278 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:53.645302 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:53.645435 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:53.648141 (+ 2706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:53.648146 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:53.648538 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:53.648750 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:53.648761 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.649828 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.649832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:53.650678 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:53.650694 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.651439 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.651444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:53.651598 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.652458 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:53.652474 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:53.653015 (+ 541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:53.655788 (+ 2773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:53.655793 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:53.655795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:53.656086 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:53.656486 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:53.656489 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:53.656492 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:53.656607 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:53.657293 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:53.657298 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:53.657460 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 00:46:53.657606 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 00:46:53.657873 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":139,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:46:53.672699 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44639
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:53.818625 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:53.819036 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:53.819089 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:53.823791 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:53.823904 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:53.823951 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:46:53.823989 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:46:53.830581 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:46:53.832355 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:53.833792 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:53.847416 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:53.847416 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:53.847553 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:53.847805 1719 server_base.cc:1034] running on GCE node
I20260927 00:46:53.848651 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:53.849759 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:53.850996 1719 hybrid_clock.cc:648] HybridClock initialized: now 1790470013850963 us; error 58 us; skew 500 ppm
Sep 27 00:46:53 dist-test-slave-86rs krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:53.855132 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:46:53.856642 1719 webserver.cc:460] Webserver started at http://127.0.98.2:32817/ using document root <none> and password file <none>
I20260927 00:46:53.857637 1719 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:53.857766 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:53.858106 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:46:53.861364 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-1/data/instance:
uuid: "c37e48fbfc574f0e8ab332020c338011"
format_stamp: "Formatted at 2026-09-27 00:46:53 on dist-test-slave-86rs"
server_key: "71c23ebe870cfb51cd691c401665f3a0"
server_key_iv: "a13051481309e1fb2b10783e7b14ff88"
server_key_version: "encryptionkey@0"
I20260927 00:46:53.862233 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c37e48fbfc574f0e8ab332020c338011"
format_stamp: "Formatted at 2026-09-27 00:46:53 on dist-test-slave-86rs"
server_key: "71c23ebe870cfb51cd691c401665f3a0"
server_key_iv: "a13051481309e1fb2b10783e7b14ff88"
server_key_version: "encryptionkey@0"
I20260927 00:46:53.867798 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 00:46:53.871430 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:53.873466 1719 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260927 00:46:53.873649 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "c37e48fbfc574f0e8ab332020c338011"
format_stamp: "Formatted at 2026-09-27 00:46:53 on dist-test-slave-86rs"
server_key: "71c23ebe870cfb51cd691c401665f3a0"
server_key_iv: "a13051481309e1fb2b10783e7b14ff88"
server_key_version: "encryptionkey@0"
I20260927 00:46:53.873828 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:53.896345 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:53.901018 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:53.901297 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:53.903388 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:46:53.903481 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:53.903575 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:46:53.903627 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:46:53 dist-test-slave-86rs krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:53.918308 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46389
I20260927 00:46:53.918390 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46389 every 8 connection(s)
I20260927 00:46:53.919579 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:46:53.921381 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.904504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41051 (local address 127.0.98.62:44639)
0927 00:46:53.904734 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:53.904742 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:53.906011 (+ 1269us) server_negotiation.cc:407] Connection header received
0927 00:46:53.907589 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:53.907598 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:53.907704 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:53.907782 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:53.910104 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.910779 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.912159 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.912558 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.912686 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.916780 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:53.916784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:53.916787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:53.916822 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:53.918745 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.919334 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.919339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.919341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.919432 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.919850 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.919855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.919857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.920197 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:53.920345 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:53.921070 (+ 725us) server_negotiation.cc:299] Negotiation successful
0927 00:46:53.921198 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 00:46:53.922533 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.905148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44639 (local address 127.0.98.2:41051)
0927 00:46:53.905768 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:53.905833 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:53.907227 (+ 1394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:53.908463 (+ 1236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:53.908475 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:53.909066 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:53.909893 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:53.909912 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.910932 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.910936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:53.912004 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:53.912015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.912760 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.912765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:53.913029 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.913849 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:53.913876 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:53.916505 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:53.918879 (+ 2374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:53.918886 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:53.918906 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:53.919203 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:53.919529 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:53.919532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:53.919534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:53.919715 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:53.920408 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:53.920419 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:53.920792 (+ 373us) client_negotiation.cc:769] Sending connection context
0927 00:46:53.920989 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 00:46:53.921341 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":172,"threads_started":1}
I20260927 00:46:53.923326 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 1719
I20260927 00:46:53.923454 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:46:53.923808 392 external_mini_cluster.cc:1428] Setting key 5be81494ad26d17be743366a3c4fd98a
I20260927 00:46:53.924217 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44639
I20260927 00:46:53.924427 1850 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:53.925236 1850 heartbeater.cc:511] Master 127.0.98.62:44639 requested a full tablet report, sending...
I20260927 00:46:53.926596 1507 ts_manager.cc:194] Registered new tserver with Master: c37e48fbfc574f0e8ab332020c338011 (127.0.98.2:46389)
I20260927 00:46:53.927349 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41051
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:46:53.938309 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.928068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55467 (local address 127.0.98.62:44639)
0927 00:46:53.928260 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:53.928266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:53.928548 (+ 282us) server_negotiation.cc:407] Connection header received
0927 00:46:53.928703 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:53.928706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:53.928759 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:53.928857 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:53.929358 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.930240 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.931231 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.931493 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.931619 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.934263 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:53.934267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:53.934269 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:53.934297 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:53.936115 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.936696 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.936702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.936704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.936815 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:53.937149 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:53.937152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:53.937155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:53.937453 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:53.937612 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:53.937936 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 00:46:53.938093 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:46:53.938323 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:53.928039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44639 (local address 127.0.98.2:55467)
0927 00:46:53.928456 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:53.928468 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:53.928603 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:53.928875 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:53.928878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:53.929120 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:53.929242 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:53.929248 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.930401 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.930405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:53.931084 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:53.931095 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:53.931634 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:53.931638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:53.931746 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:53.933646 (+ 1900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:53.933661 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:53.934081 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:53.936290 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:53.936300 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:53.936303 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:53.936570 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:53.936920 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:53.936924 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:53.936926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:53.937033 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:53.937576 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:53.937580 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:53.937718 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 00:46:53.937858 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 00:46:53.938079 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":152,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:46:53.960105 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44639
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:54.112370 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:54.113111 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:54.113204 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:54.118732 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:54.118837 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:54.118887 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:46:54.118927 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:46:54.126086 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:46:54.127655 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:54.129189 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:54.141882 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:54.141882 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:54.141882 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:54.142446 1859 server_base.cc:1034] running on GCE node
I20260927 00:46:54.143065 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:54.144047 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:54.145589 1859 hybrid_clock.cc:648] HybridClock initialized: now 1790470014145565 us; error 41 us; skew 500 ppm
Sep 27 00:46:54 dist-test-slave-86rs krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:54.150858 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:46:54.153299 1859 webserver.cc:460] Webserver started at http://127.0.98.3:42417/ using document root <none> and password file <none>
I20260927 00:46:54.154613 1859 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:54.154806 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:54.155309 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:46:54.159061 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-2/data/instance:
uuid: "0b96fde15c414bdf867471d2dfea7b0f"
format_stamp: "Formatted at 2026-09-27 00:46:54 on dist-test-slave-86rs"
server_key: "fd24df8d91f008c49a38b0684b8d364a"
server_key_iv: "f37fdd4c76bf31a5bad01be7836d7537"
server_key_version: "encryptionkey@0"
I20260927 00:46:54.159997 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0b96fde15c414bdf867471d2dfea7b0f"
format_stamp: "Formatted at 2026-09-27 00:46:54 on dist-test-slave-86rs"
server_key: "fd24df8d91f008c49a38b0684b8d364a"
server_key_iv: "f37fdd4c76bf31a5bad01be7836d7537"
server_key_version: "encryptionkey@0"
I20260927 00:46:54.167222 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260927 00:46:54.171258 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:54.173442 1859 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 00:46:54.173707 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "0b96fde15c414bdf867471d2dfea7b0f"
format_stamp: "Formatted at 2026-09-27 00:46:54 on dist-test-slave-86rs"
server_key: "fd24df8d91f008c49a38b0684b8d364a"
server_key_iv: "f37fdd4c76bf31a5bad01be7836d7537"
server_key_version: "encryptionkey@0"
I20260927 00:46:54.173910 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:54.196831 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:54.201283 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:54.201561 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:54.203521 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:46:54.203616 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:54.203693 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:46:54.203742 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:46:54 dist-test-slave-86rs krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:54.217058 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42711
I20260927 00:46:54.217384 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42711 every 8 connection(s)
I20260927 00:46:54.218513 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:46:54.220973 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 1859
I20260927 00:46:54.221114 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:46:54.221623 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.205036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45805 (local address 127.0.98.62:44639)
0927 00:46:54.205229 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:54.205235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:54.206595 (+ 1360us) server_negotiation.cc:407] Connection header received
0927 00:46:54.208378 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:54.208383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:54.208479 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:54.208594 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:54.210224 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.210979 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.212611 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.212879 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.213029 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.216448 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:54.216452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:54.216455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:54.216490 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:54.218538 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:54.219347 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:54.219351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:54.219353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:54.219459 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:54.219871 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:54.219874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:54.219876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:54.220144 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:54.220310 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:54.220994 (+ 684us) server_negotiation.cc:299] Negotiation successful
0927 00:46:54.221233 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 00:46:54.221717 392 external_mini_cluster.cc:1428] Setting key d70ef5a7bbda22eeb0129a4261a71c60
I20260927 00:46:54.222815 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.205690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44639 (local address 127.0.98.3:45805)
0927 00:46:54.206347 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:54.206438 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:54.208013 (+ 1575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:54.208860 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:54.208870 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:54.209352 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:54.210036 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:54.210053 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.211144 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.211150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:54.212455 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:54.212468 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.213106 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.213111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:54.213380 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.214249 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:54.214270 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:54.216115 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:54.218755 (+ 2640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:54.218761 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:54.218782 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:54.219212 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:54.219566 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:54.219570 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:54.219572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:54.219752 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:54.220291 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:54.220298 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:54.220659 (+ 361us) client_negotiation.cc:769] Sending connection context
0927 00:46:54.220881 (+ 222us) client_negotiation.cc:240] Negotiation successful
0927 00:46:54.221310 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":177,"threads_started":1}
I20260927 00:46:54.227368 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44639
I20260927 00:46:54.227686 1990 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:54.228425 1990 heartbeater.cc:511] Master 127.0.98.62:44639 requested a full tablet report, sending...
I20260927 00:46:54.229770 1504 ts_manager.cc:194] Registered new tserver with Master: 0b96fde15c414bdf867471d2dfea7b0f (127.0.98.3:42711)
I20260927 00:46:54.230438 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45805
I20260927 00:46:54.236938 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.227644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51961 (local address 127.0.98.62:44639)
0927 00:46:54.227790 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:54.227796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:54.228186 (+ 390us) server_negotiation.cc:407] Connection header received
0927 00:46:54.228353 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:54.228356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:54.228417 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:54.228515 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:54.229170 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.229916 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.231134 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.231450 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.231559 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.233074 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:54.233077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:54.233081 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:54.233113 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:54.234750 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:54.235228 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:54.235232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:54.235234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:54.235331 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:54.235726 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:54.235730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:54.235732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:54.235957 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:54.236126 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:54.236607 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 00:46:54.236787 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 00:46:54.236956 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.227561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44639 (local address 127.0.98.3:51961)
0927 00:46:54.228071 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:54.228088 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:54.228252 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:54.228559 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:54.228565 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:54.228850 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:54.229016 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:54.229025 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.230060 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.230063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:54.231027 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:54.231036 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.231573 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.231577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:54.231672 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.232388 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:54.232404 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:54.232910 (+ 506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:54.234873 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:54.234876 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:54.234878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:54.235127 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:54.235455 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:54.235458 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:54.235459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:54.235581 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:54.236111 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:54.236114 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:54.236291 (+ 177us) client_negotiation.cc:769] Sending connection context
0927 00:46:54.236439 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 00:46:54.236716 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":290,"threads_started":1}
I20260927 00:46:54.237419 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 00:46:54 dist-test-slave-86rs krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470014, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:46:54 dist-test-slave-86rs krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470014, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:54.269778 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.259318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49100 (local address 127.0.98.62:44639)
0927 00:46:54.259525 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:54.259529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:54.259550 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:46:54.259699 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:54.259702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:54.259764 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:54.259894 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:54.260970 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.261604 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.262652 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.262916 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.263031 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.266213 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:54.266218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:54.266221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:54.266262 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:54.267994 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:54.268550 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:54.268554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:54.268555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:54.268655 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:54.268911 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:54.268914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:54.268915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:54.269177 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:54.269294 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:54.269493 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 00:46:54.269615 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 00:46:54.274506 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:46:54.278621 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:46:54.297369 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.290108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40385 (local address 127.0.0.1:36116)
0927 00:46:54.290581 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:54.290656 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:54.290849 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:54.292070 (+ 1221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:54.292077 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:54.292127 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:54.292391 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:54.292400 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.293777 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.293780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:54.295453 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:54.295462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.296656 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.296659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:54.296805 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.296843 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 00:46:54.296950 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 00:46:54.297150 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":165,"spinlock_wait_cycles":15360,"thread_start_us":123,"threads_started":1}
I20260927 00:46:54.297435 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.290935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52348 (local address 127.0.98.2:46389)
0927 00:46:54.291386 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:54.291407 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:54.291432 (+ 25us) server_negotiation.cc:407] Connection header received
0927 00:46:54.291546 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:54.291555 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:54.291858 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:54.292374 (+ 516us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:54.292626 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.294123 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.295677 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.296407 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.296533 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.296766 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:54.297144 (+ 378us) server_negotiation.cc:299] Negotiation successful
0927 00:46:54.297261 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":175,"threads_started":1}
I20260927 00:46:54.297765 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.290494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36116 (local address 127.0.98.1:40385)
0927 00:46:54.291434 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:54.291442 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:54.291468 (+ 26us) server_negotiation.cc:407] Connection header received
0927 00:46:54.291568 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:54.291578 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:54.291859 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:54.292256 (+ 397us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:54.292488 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.293682 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.295596 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.296505 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.297251 (+ 746us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.297404 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:54.297467 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 00:46:54.297601 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":826,"thread_start_us":96,"threads_started":1}
I20260927 00:46:54.297719 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.290154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46389 (local address 127.0.0.1:52348)
0927 00:46:54.291006 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:54.291020 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:54.291121 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:54.292068 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:54.292077 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:54.292130 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:54.292538 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:54.292543 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.294259 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.294263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:54.295525 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:54.295536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.296540 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.296543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:54.296654 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.296964 (+ 310us) client_negotiation.cc:769] Sending connection context
0927 00:46:54.297043 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 00:46:54.297169 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":307,"spinlock_wait_cycles":28544,"thread_start_us":128,"threads_started":1}
I20260927 00:46:54.298966 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.291674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42711 (local address 127.0.0.1:49152)
0927 00:46:54.292014 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:54.292029 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:54.292136 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:54.292901 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:54.292903 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:54.292923 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:54.293028 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:54.293034 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.294531 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.294535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:54.295594 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:54.295604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.296900 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.296903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:54.296986 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.297005 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 00:46:54.297063 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 00:46:54.298825 (+ 1762us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":4,"spinlock_wait_cycles":31744,"thread_start_us":99,"threads_started":1}
I20260927 00:46:54.299378 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.291897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49152 (local address 127.0.98.3:42711)
0927 00:46:54.292438 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:54.292452 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:54.292476 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:46:54.292557 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:54.292562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:54.292757 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:54.292926 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:54.293127 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.294406 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.295720 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.296546 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.296648 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.296839 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:54.299089 (+ 2250us) server_negotiation.cc:299] Negotiation successful
0927 00:46:54.299226 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":149,"threads_started":1}
I20260927 00:46:54.301343 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 7a5cdc73780e440090346562f9935856 (DEFAULT_TABLE table=test-table [id=a2a5e3f2726e4005843704a2ebf69811]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:54.301322 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 7a5cdc73780e440090346562f9935856 (DEFAULT_TABLE table=test-table [id=a2a5e3f2726e4005843704a2ebf69811]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:54.302634 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a5cdc73780e440090346562f9935856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:54.302637 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a5cdc73780e440090346562f9935856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:54.302244 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 7a5cdc73780e440090346562f9935856 (DEFAULT_TABLE table=test-table [id=a2a5e3f2726e4005843704a2ebf69811]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:54.303248 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a5cdc73780e440090346562f9935856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:54.311282 2013 tablet_bootstrap.cc:492] T 7a5cdc73780e440090346562f9935856 P c37e48fbfc574f0e8ab332020c338011: Bootstrap starting.
I20260927 00:46:54.311856 2014 tablet_bootstrap.cc:492] T 7a5cdc73780e440090346562f9935856 P 9b56cc27420a49a3b5569894cbbe9b6e: Bootstrap starting.
I20260927 00:46:54.313400 2015 tablet_bootstrap.cc:492] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f: Bootstrap starting.
I20260927 00:46:54.314695 2014 tablet_bootstrap.cc:654] T 7a5cdc73780e440090346562f9935856 P 9b56cc27420a49a3b5569894cbbe9b6e: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:54.315641 2014 log.cc:824] T 7a5cdc73780e440090346562f9935856 P 9b56cc27420a49a3b5569894cbbe9b6e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:54.315665 2015 tablet_bootstrap.cc:654] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:54.316651 2015 log.cc:824] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:54.318353 2014 tablet_bootstrap.cc:492] T 7a5cdc73780e440090346562f9935856 P 9b56cc27420a49a3b5569894cbbe9b6e: No bootstrap required, opened a new log
I20260927 00:46:54.318466 2013 tablet_bootstrap.cc:654] T 7a5cdc73780e440090346562f9935856 P c37e48fbfc574f0e8ab332020c338011: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:54.318629 2014 ts_tablet_manager.cc:1397] T 7a5cdc73780e440090346562f9935856 P 9b56cc27420a49a3b5569894cbbe9b6e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 00:46:54.318701 2015 tablet_bootstrap.cc:492] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f: No bootstrap required, opened a new log
I20260927 00:46:54.318908 2015 ts_tablet_manager.cc:1397] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:46:54.319427 2013 log.cc:824] T 7a5cdc73780e440090346562f9935856 P c37e48fbfc574f0e8ab332020c338011: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:54.322101 2013 tablet_bootstrap.cc:492] T 7a5cdc73780e440090346562f9935856 P c37e48fbfc574f0e8ab332020c338011: No bootstrap required, opened a new log
I20260927 00:46:54.322352 2013 ts_tablet_manager.cc:1397] T 7a5cdc73780e440090346562f9935856 P c37e48fbfc574f0e8ab332020c338011: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260927 00:46:54.322943 2015 raft_consensus.cc:348] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } }
I20260927 00:46:54.323271 2015 raft_consensus.cc:374] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:54.323361 2015 raft_consensus.cc:729] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b96fde15c414bdf867471d2dfea7b0f, State: Initialized, Role: FOLLOWER
I20260927 00:46:54.324051 2015 consensus_queue.cc:260] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } }
I20260927 00:46:54.323968 2014 raft_consensus.cc:348] T 7a5cdc73780e440090346562f9935856 P 9b56cc27420a49a3b5569894cbbe9b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } }
I20260927 00:46:54.324312 2014 raft_consensus.cc:374] T 7a5cdc73780e440090346562f9935856 P 9b56cc27420a49a3b5569894cbbe9b6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:54.324409 2014 raft_consensus.cc:729] T 7a5cdc73780e440090346562f9935856 P 9b56cc27420a49a3b5569894cbbe9b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b56cc27420a49a3b5569894cbbe9b6e, State: Initialized, Role: FOLLOWER
I20260927 00:46:54.325068 1990 heartbeater.cc:503] Master 127.0.98.62:44639 was elected leader, sending a full tablet report...
I20260927 00:46:54.325022 2014 consensus_queue.cc:260] T 7a5cdc73780e440090346562f9935856 P 9b56cc27420a49a3b5569894cbbe9b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } }
I20260927 00:46:54.325598 2015 ts_tablet_manager.cc:1428] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260927 00:46:54.325861 1710 heartbeater.cc:503] Master 127.0.98.62:44639 was elected leader, sending a full tablet report...
I20260927 00:46:54.326248 2014 ts_tablet_manager.cc:1428] T 7a5cdc73780e440090346562f9935856 P 9b56cc27420a49a3b5569894cbbe9b6e: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260927 00:46:54.328895 2013 raft_consensus.cc:348] T 7a5cdc73780e440090346562f9935856 P c37e48fbfc574f0e8ab332020c338011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } }
I20260927 00:46:54.329339 2013 raft_consensus.cc:374] T 7a5cdc73780e440090346562f9935856 P c37e48fbfc574f0e8ab332020c338011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:54.329473 2013 raft_consensus.cc:729] T 7a5cdc73780e440090346562f9935856 P c37e48fbfc574f0e8ab332020c338011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37e48fbfc574f0e8ab332020c338011, State: Initialized, Role: FOLLOWER
I20260927 00:46:54.330075 2013 consensus_queue.cc:260] T 7a5cdc73780e440090346562f9935856 P c37e48fbfc574f0e8ab332020c338011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } }
I20260927 00:46:54.332121 2013 ts_tablet_manager.cc:1428] T 7a5cdc73780e440090346562f9935856 P c37e48fbfc574f0e8ab332020c338011: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260927 00:46:54.332609 1850 heartbeater.cc:503] Master 127.0.98.62:44639 was elected leader, sending a full tablet report...
W20260927 00:46:54.385567 1711 tablet.cc:2369] T 7a5cdc73780e440090346562f9935856 P 9b56cc27420a49a3b5569894cbbe9b6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:46:54.412213 2019 raft_consensus.cc:482] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:46:54.412582 2019 raft_consensus.cc:504] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } }
I20260927 00:46:54.414258 2019 leader_election.cc:288] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b56cc27420a49a3b5569894cbbe9b6e (127.0.98.1:40385), c37e48fbfc574f0e8ab332020c338011 (127.0.98.2:46389)
I20260927 00:46:54.418648 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.414719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54079 (local address 127.0.98.2:46389)
0927 00:46:54.414903 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:54.414909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:54.414931 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:46:54.415155 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:54.415162 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:54.415231 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:54.415304 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:54.415787 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.416449 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.417576 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.418017 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.418147 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.418227 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:54.418416 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 00:46:54.418499 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:46:54.418648 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.414598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46389 (local address 127.0.98.3:54079)
0927 00:46:54.414828 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:54.414853 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:54.414992 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:54.415453 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:54.415457 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:54.415474 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:54.415640 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:54.415648 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.416569 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.416575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:54.417433 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:54.417442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.418159 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.418163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:54.418280 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.418309 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 00:46:54.418400 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 00:46:54.418488 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260927 00:46:54.419539 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.414623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40385 (local address 127.0.98.3:59029)
0927 00:46:54.415118 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:54.415131 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:54.415221 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:54.415716 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:54.415719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:54.415738 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:54.415896 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:54.415902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.416801 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.416808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:54.418185 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:54.418194 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.419125 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.419129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:54.419219 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.419240 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:46:54.419315 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 00:46:54.419376 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":3,"thread_start_us":199,"threads_started":1}
I20260927 00:46:54.419534 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.415251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59029 (local address 127.0.98.1:40385)
0927 00:46:54.415414 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:54.415421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:54.415438 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:46:54.415508 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:54.415511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:54.415576 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:54.415703 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:54.416034 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.416656 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.418344 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.418980 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.419117 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.419194 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:54.419344 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 00:46:54.419413 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 00:46:54.419478 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a5cdc73780e440090346562f9935856" candidate_uuid: "0b96fde15c414bdf867471d2dfea7b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37e48fbfc574f0e8ab332020c338011" is_pre_election: true
I20260927 00:46:54.419914 1804 raft_consensus.cc:2454] T 7a5cdc73780e440090346562f9935856 P c37e48fbfc574f0e8ab332020c338011 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b96fde15c414bdf867471d2dfea7b0f in term 0.
I20260927 00:46:54.420430 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a5cdc73780e440090346562f9935856" candidate_uuid: "0b96fde15c414bdf867471d2dfea7b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" is_pre_election: true
I20260927 00:46:54.420506 1875 leader_election.cc:302] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b96fde15c414bdf867471d2dfea7b0f, c37e48fbfc574f0e8ab332020c338011; no voters:
I20260927 00:46:54.420827 2019 raft_consensus.cc:2781] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:46:54.420835 1664 raft_consensus.cc:2454] T 7a5cdc73780e440090346562f9935856 P 9b56cc27420a49a3b5569894cbbe9b6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b96fde15c414bdf867471d2dfea7b0f in term 0.
I20260927 00:46:54.420922 2019 raft_consensus.cc:482] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:46:54.420993 2019 raft_consensus.cc:3037] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f [term 0 FOLLOWER]: Advancing to term 1
W20260927 00:46:54.421175 1851 tablet.cc:2369] T 7a5cdc73780e440090346562f9935856 P c37e48fbfc574f0e8ab332020c338011: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:46:54.422492 2019 raft_consensus.cc:504] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } }
I20260927 00:46:54.422849 2019 leader_election.cc:288] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f [CANDIDATE]: Term 1 election: Requested vote from peers 9b56cc27420a49a3b5569894cbbe9b6e (127.0.98.1:40385), c37e48fbfc574f0e8ab332020c338011 (127.0.98.2:46389)
I20260927 00:46:54.423234 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a5cdc73780e440090346562f9935856" candidate_uuid: "0b96fde15c414bdf867471d2dfea7b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b56cc27420a49a3b5569894cbbe9b6e"
I20260927 00:46:54.423256 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a5cdc73780e440090346562f9935856" candidate_uuid: "0b96fde15c414bdf867471d2dfea7b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37e48fbfc574f0e8ab332020c338011"
I20260927 00:46:54.423384 1664 raft_consensus.cc:3037] T 7a5cdc73780e440090346562f9935856 P 9b56cc27420a49a3b5569894cbbe9b6e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:54.423421 1804 raft_consensus.cc:3037] T 7a5cdc73780e440090346562f9935856 P c37e48fbfc574f0e8ab332020c338011 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:54.424810 1664 raft_consensus.cc:2454] T 7a5cdc73780e440090346562f9935856 P 9b56cc27420a49a3b5569894cbbe9b6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b96fde15c414bdf867471d2dfea7b0f in term 1.
I20260927 00:46:54.425195 1876 leader_election.cc:302] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b96fde15c414bdf867471d2dfea7b0f, 9b56cc27420a49a3b5569894cbbe9b6e; no voters:
I20260927 00:46:54.425383 2019 raft_consensus.cc:2781] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:46:54.425648 2019 raft_consensus.cc:686] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f [term 1 LEADER]: Becoming Leader. State: Replica: 0b96fde15c414bdf867471d2dfea7b0f, State: Running, Role: LEADER
I20260927 00:46:54.425932 1804 raft_consensus.cc:2454] T 7a5cdc73780e440090346562f9935856 P c37e48fbfc574f0e8ab332020c338011 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b96fde15c414bdf867471d2dfea7b0f in term 1.
I20260927 00:46:54.426154 2019 consensus_queue.cc:237] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } }
I20260927 00:46:54.430470 1507 catalog_manager.cc:5495] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b96fde15c414bdf867471d2dfea7b0f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0b96fde15c414bdf867471d2dfea7b0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } health_report { overall_health: HEALTHY } } }
I20260927 00:46:54.464613 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.459812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49162 (local address 127.0.98.3:42711)
0927 00:46:54.459998 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:54.460010 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:54.460031 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:46:54.460411 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:54.460414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:54.460489 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:54.460601 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:46:54.460916 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.461532 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.462543 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.462803 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.462906 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.463294 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:46:54.464081 (+ 787us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:46:54.464192 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:54.464349 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 00:46:54.464429 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 00:46:54.470077 1991 tablet.cc:2369] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:46:54.472966 1804 raft_consensus.cc:1261] T 7a5cdc73780e440090346562f9935856 P c37e48fbfc574f0e8ab332020c338011 [term 1 FOLLOWER]: Refusing update from remote peer 0b96fde15c414bdf867471d2dfea7b0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:46:54.472966 1664 raft_consensus.cc:1261] T 7a5cdc73780e440090346562f9935856 P 9b56cc27420a49a3b5569894cbbe9b6e [term 1 FOLLOWER]: Refusing update from remote peer 0b96fde15c414bdf867471d2dfea7b0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:46:54.473964 2019 consensus_queue.cc:1035] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:46:54.474255 2023 consensus_queue.cc:1035] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:46:54.485630 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333765179270926336 to 7333765179093405696. Current Snapshot: MvccSnapshot[applied={T|T < 7333765179270926336}]
I20260927 00:46:54.485713 2027 mvcc.cc:205] Tried to move back new op lower bound from 7333765179270926336 to 7333765179093405696. Current Snapshot: MvccSnapshot[applied={T|T < 7333765179270926336}]
I20260927 00:46:54.506517 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:46:54.506786 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:46:54.510478 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3022093688014114898d44f408dda928: Sending DeleteTablet for 3 replicas of tablet 7a5cdc73780e440090346562f9935856
I20260927 00:46:54.511865 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a5cdc73780e440090346562f9935856 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:46:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52348
I20260927 00:46:54.511888 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a5cdc73780e440090346562f9935856 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:46:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36116
I20260927 00:46:54.511875 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a5cdc73780e440090346562f9935856 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:46:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49152
I20260927 00:46:54.512679 2040 tablet_replica.cc:330] T 7a5cdc73780e440090346562f9935856 P 9b56cc27420a49a3b5569894cbbe9b6e: stopping tablet replica
I20260927 00:46:54.512935 2038 tablet_replica.cc:330] T 7a5cdc73780e440090346562f9935856 P c37e48fbfc574f0e8ab332020c338011: stopping tablet replica
I20260927 00:46:54.513295 2040 raft_consensus.cc:2229] T 7a5cdc73780e440090346562f9935856 P 9b56cc27420a49a3b5569894cbbe9b6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:46:54.513320 2038 raft_consensus.cc:2229] T 7a5cdc73780e440090346562f9935856 P c37e48fbfc574f0e8ab332020c338011 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:46:54.513832 2040 raft_consensus.cc:2258] T 7a5cdc73780e440090346562f9935856 P 9b56cc27420a49a3b5569894cbbe9b6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:46:54.513832 2038 raft_consensus.cc:2258] T 7a5cdc73780e440090346562f9935856 P c37e48fbfc574f0e8ab332020c338011 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:46:54.514019 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 00:46:54.514556 2039 tablet_replica.cc:330] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f: stopping tablet replica
W20260927 00:46:54.514863 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:46:54.515062 2039 raft_consensus.cc:2229] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:46:54.515733 2039 raft_consensus.cc:2258] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:46:54.516168 2038 ts_tablet_manager.cc:1905] T 7a5cdc73780e440090346562f9935856 P c37e48fbfc574f0e8ab332020c338011: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:46:54.517305 2040 ts_tablet_manager.cc:1905] T 7a5cdc73780e440090346562f9935856 P 9b56cc27420a49a3b5569894cbbe9b6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:46:54.519897 2038 ts_tablet_manager.cc:1918] T 7a5cdc73780e440090346562f9935856 P c37e48fbfc574f0e8ab332020c338011: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:46:54.520103 2038 log.cc:1196] T 7a5cdc73780e440090346562f9935856 P c37e48fbfc574f0e8ab332020c338011: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-1/wal/wals/7a5cdc73780e440090346562f9935856
I20260927 00:46:54.520639 2038 ts_tablet_manager.cc:1939] T 7a5cdc73780e440090346562f9935856 P c37e48fbfc574f0e8ab332020c338011: Deleting consensus metadata
I20260927 00:46:54.520818 2040 ts_tablet_manager.cc:1918] T 7a5cdc73780e440090346562f9935856 P 9b56cc27420a49a3b5569894cbbe9b6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:46:54.521135 2040 log.cc:1196] T 7a5cdc73780e440090346562f9935856 P 9b56cc27420a49a3b5569894cbbe9b6e: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-0/wal/wals/7a5cdc73780e440090346562f9935856
I20260927 00:46:54.521350 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 2225320e8a6e42bd9338dd8ca82a97b6 (DEFAULT_TABLE table=test-table [id=3fe4082e441e43d09536f72b6a686050]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:54.521678 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2225320e8a6e42bd9338dd8ca82a97b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:54.521724 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 2225320e8a6e42bd9338dd8ca82a97b6 (DEFAULT_TABLE table=test-table [id=3fe4082e441e43d09536f72b6a686050]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:54.521873 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 2225320e8a6e42bd9338dd8ca82a97b6 (DEFAULT_TABLE table=test-table [id=3fe4082e441e43d09536f72b6a686050]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:54.522017 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2225320e8a6e42bd9338dd8ca82a97b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:54.523674 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2225320e8a6e42bd9338dd8ca82a97b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:54.525107 2014 tablet_bootstrap.cc:492] T 2225320e8a6e42bd9338dd8ca82a97b6 P 9b56cc27420a49a3b5569894cbbe9b6e: Bootstrap starting.
I20260927 00:46:54.526333 2013 tablet_bootstrap.cc:492] T 2225320e8a6e42bd9338dd8ca82a97b6 P c37e48fbfc574f0e8ab332020c338011: Bootstrap starting.
I20260927 00:46:54.526433 2014 tablet_bootstrap.cc:654] T 2225320e8a6e42bd9338dd8ca82a97b6 P 9b56cc27420a49a3b5569894cbbe9b6e: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:54.527590 2013 tablet_bootstrap.cc:654] T 2225320e8a6e42bd9338dd8ca82a97b6 P c37e48fbfc574f0e8ab332020c338011: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:54.528714 2015 tablet_bootstrap.cc:492] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f: Bootstrap starting.
I20260927 00:46:54.529194 1491 catalog_manager.cc:4841] TS c37e48fbfc574f0e8ab332020c338011 (127.0.98.2:46389): tablet 7a5cdc73780e440090346562f9935856 (table test-table [id=a2a5e3f2726e4005843704a2ebf69811]) successfully deleted
I20260927 00:46:54.529727 2013 tablet_bootstrap.cc:492] T 2225320e8a6e42bd9338dd8ca82a97b6 P c37e48fbfc574f0e8ab332020c338011: No bootstrap required, opened a new log
I20260927 00:46:54.529819 2013 ts_tablet_manager.cc:1397] T 2225320e8a6e42bd9338dd8ca82a97b6 P c37e48fbfc574f0e8ab332020c338011: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 00:46:54.529979 2015 tablet_bootstrap.cc:654] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:54.530426 2013 raft_consensus.cc:348] T 2225320e8a6e42bd9338dd8ca82a97b6 P c37e48fbfc574f0e8ab332020c338011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } }
I20260927 00:46:54.530588 2013 raft_consensus.cc:374] T 2225320e8a6e42bd9338dd8ca82a97b6 P c37e48fbfc574f0e8ab332020c338011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:54.530629 2013 raft_consensus.cc:729] T 2225320e8a6e42bd9338dd8ca82a97b6 P c37e48fbfc574f0e8ab332020c338011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37e48fbfc574f0e8ab332020c338011, State: Initialized, Role: FOLLOWER
I20260927 00:46:54.530803 2013 consensus_queue.cc:260] T 2225320e8a6e42bd9338dd8ca82a97b6 P c37e48fbfc574f0e8ab332020c338011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } }
I20260927 00:46:54.531205 2013 ts_tablet_manager.cc:1428] T 2225320e8a6e42bd9338dd8ca82a97b6 P c37e48fbfc574f0e8ab332020c338011: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 00:46:54.531584 2015 tablet_bootstrap.cc:492] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f: No bootstrap required, opened a new log
I20260927 00:46:54.531672 2015 ts_tablet_manager.cc:1397] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260927 00:46:54.532203 2039 ts_tablet_manager.cc:1905] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:46:54.532259 2015 raft_consensus.cc:348] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } }
I20260927 00:46:54.533017 2040 ts_tablet_manager.cc:1939] T 7a5cdc73780e440090346562f9935856 P 9b56cc27420a49a3b5569894cbbe9b6e: Deleting consensus metadata
I20260927 00:46:54.533062 2015 raft_consensus.cc:374] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:54.533103 2015 raft_consensus.cc:729] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b96fde15c414bdf867471d2dfea7b0f, State: Initialized, Role: FOLLOWER
I20260927 00:46:54.533295 2015 consensus_queue.cc:260] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } }
I20260927 00:46:54.533654 2015 ts_tablet_manager.cc:1428] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260927 00:46:54.535782 2014 tablet_bootstrap.cc:492] T 2225320e8a6e42bd9338dd8ca82a97b6 P 9b56cc27420a49a3b5569894cbbe9b6e: No bootstrap required, opened a new log
I20260927 00:46:54.535928 2014 ts_tablet_manager.cc:1397] T 2225320e8a6e42bd9338dd8ca82a97b6 P 9b56cc27420a49a3b5569894cbbe9b6e: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.001s
I20260927 00:46:54.536273 2039 ts_tablet_manager.cc:1918] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:46:54.536392 2039 log.cc:1196] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-2/wal/wals/7a5cdc73780e440090346562f9935856
I20260927 00:46:54.536767 2039 ts_tablet_manager.cc:1939] T 7a5cdc73780e440090346562f9935856 P 0b96fde15c414bdf867471d2dfea7b0f: Deleting consensus metadata
I20260927 00:46:54.536767 2014 raft_consensus.cc:348] T 2225320e8a6e42bd9338dd8ca82a97b6 P 9b56cc27420a49a3b5569894cbbe9b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } }
I20260927 00:46:54.536983 2014 raft_consensus.cc:374] T 2225320e8a6e42bd9338dd8ca82a97b6 P 9b56cc27420a49a3b5569894cbbe9b6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:54.537022 2014 raft_consensus.cc:729] T 2225320e8a6e42bd9338dd8ca82a97b6 P 9b56cc27420a49a3b5569894cbbe9b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b56cc27420a49a3b5569894cbbe9b6e, State: Initialized, Role: FOLLOWER
I20260927 00:46:54.537210 2014 consensus_queue.cc:260] T 2225320e8a6e42bd9338dd8ca82a97b6 P 9b56cc27420a49a3b5569894cbbe9b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } }
I20260927 00:46:54.537622 2014 ts_tablet_manager.cc:1428] T 2225320e8a6e42bd9338dd8ca82a97b6 P 9b56cc27420a49a3b5569894cbbe9b6e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:46:54.538587 2035 raft_consensus.cc:482] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:46:54.538576 1491 catalog_manager.cc:4841] TS 0b96fde15c414bdf867471d2dfea7b0f (127.0.98.3:42711): tablet 7a5cdc73780e440090346562f9935856 (table test-table [id=a2a5e3f2726e4005843704a2ebf69811]) successfully deleted
I20260927 00:46:54.538698 2035 raft_consensus.cc:504] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } }
I20260927 00:46:54.539214 2035 leader_election.cc:288] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b56cc27420a49a3b5569894cbbe9b6e (127.0.98.1:40385), c37e48fbfc574f0e8ab332020c338011 (127.0.98.2:46389)
I20260927 00:46:54.539731 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2225320e8a6e42bd9338dd8ca82a97b6" candidate_uuid: "0b96fde15c414bdf867471d2dfea7b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37e48fbfc574f0e8ab332020c338011" is_pre_election: true
I20260927 00:46:54.539891 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2225320e8a6e42bd9338dd8ca82a97b6" candidate_uuid: "0b96fde15c414bdf867471d2dfea7b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" is_pre_election: true
I20260927 00:46:54.539927 1804 raft_consensus.cc:2454] T 2225320e8a6e42bd9338dd8ca82a97b6 P c37e48fbfc574f0e8ab332020c338011 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b96fde15c414bdf867471d2dfea7b0f in term 0.
I20260927 00:46:54.540026 1664 raft_consensus.cc:2454] T 2225320e8a6e42bd9338dd8ca82a97b6 P 9b56cc27420a49a3b5569894cbbe9b6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b96fde15c414bdf867471d2dfea7b0f in term 0.
I20260927 00:46:54.540369 1876 leader_election.cc:302] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b96fde15c414bdf867471d2dfea7b0f, 9b56cc27420a49a3b5569894cbbe9b6e; no voters:
I20260927 00:46:54.540650 2035 raft_consensus.cc:2781] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:46:54.540719 2035 raft_consensus.cc:482] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:46:54.540699 1492 catalog_manager.cc:4841] TS 9b56cc27420a49a3b5569894cbbe9b6e (127.0.98.1:40385): tablet 7a5cdc73780e440090346562f9935856 (table test-table [id=a2a5e3f2726e4005843704a2ebf69811]) successfully deleted
I20260927 00:46:54.540766 2035 raft_consensus.cc:3037] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:54.542344 2035 raft_consensus.cc:504] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } }
I20260927 00:46:54.542794 2035 leader_election.cc:288] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f [CANDIDATE]: Term 1 election: Requested vote from peers 9b56cc27420a49a3b5569894cbbe9b6e (127.0.98.1:40385), c37e48fbfc574f0e8ab332020c338011 (127.0.98.2:46389)
I20260927 00:46:54.543129 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2225320e8a6e42bd9338dd8ca82a97b6" candidate_uuid: "0b96fde15c414bdf867471d2dfea7b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b56cc27420a49a3b5569894cbbe9b6e"
I20260927 00:46:54.543268 1664 raft_consensus.cc:3037] T 2225320e8a6e42bd9338dd8ca82a97b6 P 9b56cc27420a49a3b5569894cbbe9b6e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:54.543339 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2225320e8a6e42bd9338dd8ca82a97b6" candidate_uuid: "0b96fde15c414bdf867471d2dfea7b0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37e48fbfc574f0e8ab332020c338011"
I20260927 00:46:54.543491 1804 raft_consensus.cc:3037] T 2225320e8a6e42bd9338dd8ca82a97b6 P c37e48fbfc574f0e8ab332020c338011 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:54.544651 1804 raft_consensus.cc:2454] T 2225320e8a6e42bd9338dd8ca82a97b6 P c37e48fbfc574f0e8ab332020c338011 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b96fde15c414bdf867471d2dfea7b0f in term 1.
I20260927 00:46:54.544694 1664 raft_consensus.cc:2454] T 2225320e8a6e42bd9338dd8ca82a97b6 P 9b56cc27420a49a3b5569894cbbe9b6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b96fde15c414bdf867471d2dfea7b0f in term 1.
I20260927 00:46:54.545030 1876 leader_election.cc:302] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b96fde15c414bdf867471d2dfea7b0f, 9b56cc27420a49a3b5569894cbbe9b6e; no voters:
I20260927 00:46:54.545236 2035 raft_consensus.cc:2781] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:46:54.545361 2035 raft_consensus.cc:686] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f [term 1 LEADER]: Becoming Leader. State: Replica: 0b96fde15c414bdf867471d2dfea7b0f, State: Running, Role: LEADER
I20260927 00:46:54.545521 2035 consensus_queue.cc:237] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } }
I20260927 00:46:54.547207 1504 catalog_manager.cc:5495] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b96fde15c414bdf867471d2dfea7b0f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0b96fde15c414bdf867471d2dfea7b0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } health_report { overall_health: HEALTHY } } }
Sep 27 00:46:54 dist-test-slave-86rs krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:54.597908 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.586800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49114 (local address 127.0.98.62:44639)
0927 00:46:54.586953 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:54.586958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:54.587240 (+ 282us) server_negotiation.cc:407] Connection header received
0927 00:46:54.587419 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:54.587422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:54.587477 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:54.587580 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:54.588527 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.589413 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.590295 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.590653 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.590758 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.593928 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:54.593932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:54.593935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:54.593968 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:54.595704 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:54.596356 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:54.596362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:54.596370 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:54.596461 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:54.596788 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:54.596790 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:54.596791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:54.597006 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:54.597086 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:54.597479 (+ 393us) server_negotiation.cc:299] Negotiation successful
0927 00:46:54.597731 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":111,"server-negotiator.queue_time_us":52}
I20260927 00:46:54.597911 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.586705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44639 (local address 127.0.0.1:49114)
0927 00:46:54.587136 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:54.587152 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:54.587315 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:54.587613 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:54.587615 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:54.588272 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:54.588393 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:54.588401 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.589548 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.589551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:54.590176 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:54.590184 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.590782 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.590785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:54.590852 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.591592 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:54.591604 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:54.593695 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:54.595823 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:54.595837 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:54.595847 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:54.596216 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:54.596563 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:54.596568 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:54.596572 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:54.596684 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:54.597116 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:54.597125 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:54.597286 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 00:46:54.597411 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 00:46:54.597638 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":147,"threads_started":1}
I20260927 00:46:54.604441 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49114:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 00:46:54.605675 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:46:54.612509 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 4c5fb76444ff492090cc4a60f72f1db4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bfbdf2c3b72448e283352932a236a85f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:46:54.612882 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 4c5fb76444ff492090cc4a60f72f1db4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bfbdf2c3b72448e283352932a236a85f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:46:54.613171 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c5fb76444ff492090cc4a60f72f1db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:54.613336 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c5fb76444ff492090cc4a60f72f1db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:54.616874 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 4c5fb76444ff492090cc4a60f72f1db4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bfbdf2c3b72448e283352932a236a85f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:46:54.617233 2013 tablet_bootstrap.cc:492] T 4c5fb76444ff492090cc4a60f72f1db4 P c37e48fbfc574f0e8ab332020c338011: Bootstrap starting.
I20260927 00:46:54.617348 2014 tablet_bootstrap.cc:492] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e: Bootstrap starting.
I20260927 00:46:54.617444 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c5fb76444ff492090cc4a60f72f1db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:54.618538 2014 tablet_bootstrap.cc:654] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:54.618538 2013 tablet_bootstrap.cc:654] T 4c5fb76444ff492090cc4a60f72f1db4 P c37e48fbfc574f0e8ab332020c338011: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:54.620719 2013 tablet_bootstrap.cc:492] T 4c5fb76444ff492090cc4a60f72f1db4 P c37e48fbfc574f0e8ab332020c338011: No bootstrap required, opened a new log
I20260927 00:46:54.620728 2014 tablet_bootstrap.cc:492] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e: No bootstrap required, opened a new log
I20260927 00:46:54.620831 2014 ts_tablet_manager.cc:1397] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 00:46:54.620831 2013 ts_tablet_manager.cc:1397] T 4c5fb76444ff492090cc4a60f72f1db4 P c37e48fbfc574f0e8ab332020c338011: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 00:46:54.621351 2015 tablet_bootstrap.cc:492] T 4c5fb76444ff492090cc4a60f72f1db4 P 0b96fde15c414bdf867471d2dfea7b0f: Bootstrap starting.
I20260927 00:46:54.621371 2013 raft_consensus.cc:348] T 4c5fb76444ff492090cc4a60f72f1db4 P c37e48fbfc574f0e8ab332020c338011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } }
I20260927 00:46:54.621371 2014 raft_consensus.cc:348] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } }
I20260927 00:46:54.621500 2013 raft_consensus.cc:374] T 4c5fb76444ff492090cc4a60f72f1db4 P c37e48fbfc574f0e8ab332020c338011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:54.621515 2014 raft_consensus.cc:374] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:54.621539 2013 raft_consensus.cc:729] T 4c5fb76444ff492090cc4a60f72f1db4 P c37e48fbfc574f0e8ab332020c338011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37e48fbfc574f0e8ab332020c338011, State: Initialized, Role: FOLLOWER
I20260927 00:46:54.621551 2014 raft_consensus.cc:729] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b56cc27420a49a3b5569894cbbe9b6e, State: Initialized, Role: FOLLOWER
I20260927 00:46:54.621690 2013 consensus_queue.cc:260] T 4c5fb76444ff492090cc4a60f72f1db4 P c37e48fbfc574f0e8ab332020c338011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } }
I20260927 00:46:54.621690 2014 consensus_queue.cc:260] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } }
I20260927 00:46:54.621990 2013 ts_tablet_manager.cc:1428] T 4c5fb76444ff492090cc4a60f72f1db4 P c37e48fbfc574f0e8ab332020c338011: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:46:54.622020 2014 ts_tablet_manager.cc:1428] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 00:46:54.622291 2020 tablet_replica.cc:439] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } }
I20260927 00:46:54.622457 2020 tablet_replica.cc:442] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:46:54.622462 2015 tablet_bootstrap.cc:654] T 4c5fb76444ff492090cc4a60f72f1db4 P 0b96fde15c414bdf867471d2dfea7b0f: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:54.623291 2021 tablet_replica.cc:439] T 4c5fb76444ff492090cc4a60f72f1db4 P c37e48fbfc574f0e8ab332020c338011: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } }
I20260927 00:46:54.623467 2021 tablet_replica.cc:442] T 4c5fb76444ff492090cc4a60f72f1db4 P c37e48fbfc574f0e8ab332020c338011: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:46:54.625413 2015 tablet_bootstrap.cc:492] T 4c5fb76444ff492090cc4a60f72f1db4 P 0b96fde15c414bdf867471d2dfea7b0f: No bootstrap required, opened a new log
I20260927 00:46:54.625559 2015 ts_tablet_manager.cc:1397] T 4c5fb76444ff492090cc4a60f72f1db4 P 0b96fde15c414bdf867471d2dfea7b0f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260927 00:46:54.626250 2015 raft_consensus.cc:348] T 4c5fb76444ff492090cc4a60f72f1db4 P 0b96fde15c414bdf867471d2dfea7b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } }
I20260927 00:46:54.626407 2015 raft_consensus.cc:374] T 4c5fb76444ff492090cc4a60f72f1db4 P 0b96fde15c414bdf867471d2dfea7b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:54.626446 2015 raft_consensus.cc:729] T 4c5fb76444ff492090cc4a60f72f1db4 P 0b96fde15c414bdf867471d2dfea7b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b96fde15c414bdf867471d2dfea7b0f, State: Initialized, Role: FOLLOWER
I20260927 00:46:54.626741 2015 consensus_queue.cc:260] T 4c5fb76444ff492090cc4a60f72f1db4 P 0b96fde15c414bdf867471d2dfea7b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } }
I20260927 00:46:54.627100 2035 tablet_replica.cc:439] T 4c5fb76444ff492090cc4a60f72f1db4 P 0b96fde15c414bdf867471d2dfea7b0f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } }
I20260927 00:46:54.627305 2035 tablet_replica.cc:442] T 4c5fb76444ff492090cc4a60f72f1db4 P 0b96fde15c414bdf867471d2dfea7b0f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:46:54.627205 2015 ts_tablet_manager.cc:1428] T 4c5fb76444ff492090cc4a60f72f1db4 P 0b96fde15c414bdf867471d2dfea7b0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 00:46:54.771060 2020 raft_consensus.cc:482] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:46:54.771287 2020 raft_consensus.cc:504] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } }
I20260927 00:46:54.772930 2020 leader_election.cc:288] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c37e48fbfc574f0e8ab332020c338011 (127.0.98.2:46389), 0b96fde15c414bdf867471d2dfea7b0f (127.0.98.3:42711)
I20260927 00:46:54.777433 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.773338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46389 (local address 127.0.98.1:54885)
0927 00:46:54.773716 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:54.773734 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:54.773885 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:54.774300 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:54.774303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:54.774325 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:54.774459 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:54.774465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.775278 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.775284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:54.776205 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:54.776214 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.776944 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.776948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:54.777063 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.777094 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 00:46:54.777180 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 00:46:54.777258 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":174,"threads_started":1}
I20260927 00:46:54.777422 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.773456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54885 (local address 127.0.98.2:46389)
0927 00:46:54.773654 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:54.773658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:54.773847 (+ 189us) server_negotiation.cc:407] Connection header received
0927 00:46:54.774107 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:54.774111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:54.774177 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:54.774284 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:54.774604 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.775159 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.776385 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.776821 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.776943 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.777030 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:54.777196 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 00:46:54.777272 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 00:46:54.777884 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.774061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55941 (local address 127.0.98.3:42711)
0927 00:46:54.774235 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:54.774241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:54.774393 (+ 152us) server_negotiation.cc:407] Connection header received
0927 00:46:54.774609 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:54.774612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:54.774662 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:54.774763 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:54.775051 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.775643 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.776815 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.777243 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.777373 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.777450 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:54.777651 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 00:46:54.777740 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 00:46:54.777992 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c5fb76444ff492090cc4a60f72f1db4" candidate_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37e48fbfc574f0e8ab332020c338011" is_pre_election: true
I20260927 00:46:54.778214 1804 raft_consensus.cc:2454] T 4c5fb76444ff492090cc4a60f72f1db4 P c37e48fbfc574f0e8ab332020c338011 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b56cc27420a49a3b5569894cbbe9b6e in term 0.
I20260927 00:46:54.778366 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.773861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42711 (local address 127.0.98.1:55941)
0927 00:46:54.774259 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:54.774277 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:54.774443 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:54.774773 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:54.774777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:54.774795 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:54.774917 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:54.774927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.775779 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.775784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:54.776666 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:54.776676 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.777375 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.777379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:54.777476 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.777505 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 00:46:54.778093 (+ 588us) client_negotiation.cc:240] Negotiation successful
0927 00:46:54.778181 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":127,"threads_started":1}
I20260927 00:46:54.778684 1595 leader_election.cc:302] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b56cc27420a49a3b5569894cbbe9b6e, c37e48fbfc574f0e8ab332020c338011; no voters:
I20260927 00:46:54.779062 2020 raft_consensus.cc:2781] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:46:54.779156 2020 raft_consensus.cc:482] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:46:54.779201 2020 raft_consensus.cc:3037] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:54.779457 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c5fb76444ff492090cc4a60f72f1db4" candidate_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b96fde15c414bdf867471d2dfea7b0f" is_pre_election: true
I20260927 00:46:54.779793 1944 raft_consensus.cc:2454] T 4c5fb76444ff492090cc4a60f72f1db4 P 0b96fde15c414bdf867471d2dfea7b0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b56cc27420a49a3b5569894cbbe9b6e in term 0.
I20260927 00:46:54.780761 2020 raft_consensus.cc:504] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } }
I20260927 00:46:54.781152 2020 leader_election.cc:288] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e [CANDIDATE]: Term 1 election: Requested vote from peers c37e48fbfc574f0e8ab332020c338011 (127.0.98.2:46389), 0b96fde15c414bdf867471d2dfea7b0f (127.0.98.3:42711)
I20260927 00:46:54.781679 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c5fb76444ff492090cc4a60f72f1db4" candidate_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37e48fbfc574f0e8ab332020c338011"
I20260927 00:46:54.781714 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c5fb76444ff492090cc4a60f72f1db4" candidate_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b96fde15c414bdf867471d2dfea7b0f"
I20260927 00:46:54.781837 1804 raft_consensus.cc:3037] T 4c5fb76444ff492090cc4a60f72f1db4 P c37e48fbfc574f0e8ab332020c338011 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:54.781836 1944 raft_consensus.cc:3037] T 4c5fb76444ff492090cc4a60f72f1db4 P 0b96fde15c414bdf867471d2dfea7b0f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:54.783231 1804 raft_consensus.cc:2454] T 4c5fb76444ff492090cc4a60f72f1db4 P c37e48fbfc574f0e8ab332020c338011 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b56cc27420a49a3b5569894cbbe9b6e in term 1.
I20260927 00:46:54.783303 1944 raft_consensus.cc:2454] T 4c5fb76444ff492090cc4a60f72f1db4 P 0b96fde15c414bdf867471d2dfea7b0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b56cc27420a49a3b5569894cbbe9b6e in term 1.
I20260927 00:46:54.783567 1595 leader_election.cc:302] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b56cc27420a49a3b5569894cbbe9b6e, c37e48fbfc574f0e8ab332020c338011; no voters:
I20260927 00:46:54.783793 2020 raft_consensus.cc:2781] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:46:54.784094 2020 raft_consensus.cc:686] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e [term 1 LEADER]: Becoming Leader. State: Replica: 9b56cc27420a49a3b5569894cbbe9b6e, State: Running, Role: LEADER
I20260927 00:46:54.784468 2020 consensus_queue.cc:237] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } }
I20260927 00:46:54.786103 2056 tablet_replica.cc:439] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e: TxnStatusTablet state changed. Reason: New leader 9b56cc27420a49a3b5569894cbbe9b6e. Latest consensus state: current_term: 1 leader_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } }
I20260927 00:46:54.786247 2056 tablet_replica.cc:442] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:46:54.786610 2058 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 00:46:54.786825 2058 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 00:46:54.787623 1503 catalog_manager.cc:5495] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b56cc27420a49a3b5569894cbbe9b6e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } health_report { overall_health: UNKNOWN } } }
I20260927 00:46:54.834088 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.829118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40385 (local address 127.0.0.1:36122)
0927 00:46:54.829318 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:54.829338 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:54.829472 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:54.829892 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:54.829895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:54.829907 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:46:54.830042 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:54.830049 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.831160 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.831164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:54.831848 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:54.831857 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.832512 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.832514 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:54.832599 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.832658 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:46:54.833740 (+ 1082us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:46:54.833763 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:46:54.833825 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 00:46:54.833926 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 00:46:54.834084 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.829306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36122 (local address 127.0.98.1:40385)
0927 00:46:54.829468 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:54.829473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:54.829492 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:46:54.829627 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:54.829631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:54.829718 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:54.829829 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:46:54.830180 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.831028 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.831984 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.832396 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.832500 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.832851 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:46:54.833619 (+ 768us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:46:54.833759 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:54.833842 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 00:46:54.833915 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 00:46:54.838210 1804 raft_consensus.cc:1261] T 4c5fb76444ff492090cc4a60f72f1db4 P c37e48fbfc574f0e8ab332020c338011 [term 1 FOLLOWER]: Refusing update from remote peer 9b56cc27420a49a3b5569894cbbe9b6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:46:54.838506 1944 raft_consensus.cc:1261] T 4c5fb76444ff492090cc4a60f72f1db4 P 0b96fde15c414bdf867471d2dfea7b0f [term 1 FOLLOWER]: Refusing update from remote peer 9b56cc27420a49a3b5569894cbbe9b6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:46:54.838850 2020 consensus_queue.cc:1035] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:46:54.839094 2056 consensus_queue.cc:1035] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:46:54.840888 2021 tablet_replica.cc:439] T 4c5fb76444ff492090cc4a60f72f1db4 P c37e48fbfc574f0e8ab332020c338011: TxnStatusTablet state changed. Reason: New leader 9b56cc27420a49a3b5569894cbbe9b6e. Latest consensus state: current_term: 1 leader_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } }
I20260927 00:46:54.841117 2021 tablet_replica.cc:442] T 4c5fb76444ff492090cc4a60f72f1db4 P c37e48fbfc574f0e8ab332020c338011: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:46:54.841142 2035 tablet_replica.cc:439] T 4c5fb76444ff492090cc4a60f72f1db4 P 0b96fde15c414bdf867471d2dfea7b0f: TxnStatusTablet state changed. Reason: New leader 9b56cc27420a49a3b5569894cbbe9b6e. Latest consensus state: current_term: 1 leader_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } }
I20260927 00:46:54.841269 2035 tablet_replica.cc:442] T 4c5fb76444ff492090cc4a60f72f1db4 P 0b96fde15c414bdf867471d2dfea7b0f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:46:54.843967 2020 tablet_replica.cc:439] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } }
I20260927 00:46:54.844122 2020 tablet_replica.cc:442] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:46:54.844767 2035 tablet_replica.cc:439] T 4c5fb76444ff492090cc4a60f72f1db4 P 0b96fde15c414bdf867471d2dfea7b0f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } }
I20260927 00:46:54.844903 2035 tablet_replica.cc:442] T 4c5fb76444ff492090cc4a60f72f1db4 P 0b96fde15c414bdf867471d2dfea7b0f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:46:54.847419 2020 tablet_replica.cc:439] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } }
I20260927 00:46:54.847584 2020 tablet_replica.cc:442] T 4c5fb76444ff492090cc4a60f72f1db4 P 9b56cc27420a49a3b5569894cbbe9b6e: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:46:54.848143 2021 tablet_replica.cc:439] T 4c5fb76444ff492090cc4a60f72f1db4 P c37e48fbfc574f0e8ab332020c338011: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 } } peers { permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 } } peers { permanent_uuid: "0b96fde15c414bdf867471d2dfea7b0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } }
I20260927 00:46:54.848318 2021 tablet_replica.cc:442] T 4c5fb76444ff492090cc4a60f72f1db4 P c37e48fbfc574f0e8ab332020c338011: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:46:54.865815 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.861372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40385 (local address 127.0.98.3:48415)
0927 00:46:54.861744 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:54.861768 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:54.861898 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:54.862240 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:54.862243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:54.862253 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:46:54.862379 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:54.862384 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.863496 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.863499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:54.864302 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:54.864314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.864937 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.864939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:54.865012 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.865087 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:46:54.865455 (+ 368us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:46:54.865462 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:46:54.865558 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 00:46:54.865649 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":130,"threads_started":1}
I20260927 00:46:54.865823 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.861503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48415 (local address 127.0.98.1:40385)
0927 00:46:54.861677 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:54.861683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:54.861862 (+ 179us) server_negotiation.cc:407] Connection header received
0927 00:46:54.862022 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:54.862028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:54.862096 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:54.862191 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:46:54.862506 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.863361 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.864435 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.864801 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.864917 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.865239 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:46:54.865341 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:46:54.865416 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:54.865571 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 00:46:54.865639 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 00:46:54.875152 1804 raft_consensus.cc:1261] T 2225320e8a6e42bd9338dd8ca82a97b6 P c37e48fbfc574f0e8ab332020c338011 [term 1 FOLLOWER]: Refusing update from remote peer 0b96fde15c414bdf867471d2dfea7b0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:46:54.875152 1664 raft_consensus.cc:1261] T 2225320e8a6e42bd9338dd8ca82a97b6 P 9b56cc27420a49a3b5569894cbbe9b6e [term 1 FOLLOWER]: Refusing update from remote peer 0b96fde15c414bdf867471d2dfea7b0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:46:54.875653 2031 consensus_queue.cc:1035] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b56cc27420a49a3b5569894cbbe9b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:46:54.875880 2035 consensus_queue.cc:1035] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37e48fbfc574f0e8ab332020c338011" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:46:54.879295 2026 mvcc.cc:205] Tried to move back new op lower bound from 7333765180924342272 to 7333765179579916288. Current Snapshot: MvccSnapshot[applied={T|T < 7333765180924342272}]
I20260927 00:46:54.906105 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.901357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53709 (local address 127.0.98.3:42711)
0927 00:46:54.901489 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:54.901494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:54.901824 (+ 330us) server_negotiation.cc:407] Connection header received
0927 00:46:54.902049 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:54.902053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:54.902120 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:54.902173 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:46:54.902574 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.903363 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.904718 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.905018 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.905150 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.905485 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:46:54.905595 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:46:54.905679 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:54.905854 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 00:46:54.905930 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:46:54.906637 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.901286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42711 (local address 127.0.98.1:53709)
0927 00:46:54.901731 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:54.901749 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:54.901916 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:54.902326 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:54.902329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:54.902338 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:46:54.902458 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:54.902464 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.903507 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.903510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:54.904558 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:54.904570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.905146 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.905151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:54.905245 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.905313 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:46:54.905748 (+ 435us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:46:54.905754 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:46:54.906370 (+ 616us) client_negotiation.cc:240] Negotiation successful
0927 00:46:54.906479 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":135,"threads_started":1}
I20260927 00:46:54.945724 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:46:54.945961 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:46:54.948738 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3022093688014114898d44f408dda928: Sending DeleteTablet for 3 replicas of tablet 2225320e8a6e42bd9338dd8ca82a97b6
I20260927 00:46:54.949632 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 2225320e8a6e42bd9338dd8ca82a97b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:46:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36116
I20260927 00:46:54.949671 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 2225320e8a6e42bd9338dd8ca82a97b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:46:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49152
I20260927 00:46:54.949790 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 2225320e8a6e42bd9338dd8ca82a97b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:46:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52348
I20260927 00:46:54.949936 2039 tablet_replica.cc:330] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f: stopping tablet replica
I20260927 00:46:54.949998 2040 tablet_replica.cc:330] T 2225320e8a6e42bd9338dd8ca82a97b6 P 9b56cc27420a49a3b5569894cbbe9b6e: stopping tablet replica
I20260927 00:46:54.950111 2040 raft_consensus.cc:2229] T 2225320e8a6e42bd9338dd8ca82a97b6 P 9b56cc27420a49a3b5569894cbbe9b6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:46:54.950111 2039 raft_consensus.cc:2229] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:46:54.950263 2040 raft_consensus.cc:2258] T 2225320e8a6e42bd9338dd8ca82a97b6 P 9b56cc27420a49a3b5569894cbbe9b6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:46:54.950390 2039 raft_consensus.cc:2258] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:46:54.950613 2038 tablet_replica.cc:330] T 2225320e8a6e42bd9338dd8ca82a97b6 P c37e48fbfc574f0e8ab332020c338011: stopping tablet replica
I20260927 00:46:54.950758 2038 raft_consensus.cc:2229] T 2225320e8a6e42bd9338dd8ca82a97b6 P c37e48fbfc574f0e8ab332020c338011 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:46:54.950896 2038 raft_consensus.cc:2258] T 2225320e8a6e42bd9338dd8ca82a97b6 P c37e48fbfc574f0e8ab332020c338011 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:46:54.951742 2040 ts_tablet_manager.cc:1905] T 2225320e8a6e42bd9338dd8ca82a97b6 P 9b56cc27420a49a3b5569894cbbe9b6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:46:54.951851 2039 ts_tablet_manager.cc:1905] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:46:54.951951 2038 ts_tablet_manager.cc:1905] T 2225320e8a6e42bd9338dd8ca82a97b6 P c37e48fbfc574f0e8ab332020c338011: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:46:54.956400 2040 ts_tablet_manager.cc:1918] T 2225320e8a6e42bd9338dd8ca82a97b6 P 9b56cc27420a49a3b5569894cbbe9b6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:46:54.956400 2039 ts_tablet_manager.cc:1918] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:46:54.956441 2038 ts_tablet_manager.cc:1918] T 2225320e8a6e42bd9338dd8ca82a97b6 P c37e48fbfc574f0e8ab332020c338011: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:46:54.956543 2039 log.cc:1196] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-2/wal/wals/2225320e8a6e42bd9338dd8ca82a97b6
I20260927 00:46:54.956543 2040 log.cc:1196] T 2225320e8a6e42bd9338dd8ca82a97b6 P 9b56cc27420a49a3b5569894cbbe9b6e: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-0/wal/wals/2225320e8a6e42bd9338dd8ca82a97b6
I20260927 00:46:54.956598 2038 log.cc:1196] T 2225320e8a6e42bd9338dd8ca82a97b6 P c37e48fbfc574f0e8ab332020c338011: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790470006085796-392-0/minicluster-data/ts-1/wal/wals/2225320e8a6e42bd9338dd8ca82a97b6
I20260927 00:46:54.956871 2039 ts_tablet_manager.cc:1939] T 2225320e8a6e42bd9338dd8ca82a97b6 P 0b96fde15c414bdf867471d2dfea7b0f: Deleting consensus metadata
I20260927 00:46:54.956871 2040 ts_tablet_manager.cc:1939] T 2225320e8a6e42bd9338dd8ca82a97b6 P 9b56cc27420a49a3b5569894cbbe9b6e: Deleting consensus metadata
I20260927 00:46:54.956991 2038 ts_tablet_manager.cc:1939] T 2225320e8a6e42bd9338dd8ca82a97b6 P c37e48fbfc574f0e8ab332020c338011: Deleting consensus metadata
I20260927 00:46:54.958264 1491 catalog_manager.cc:4841] TS 0b96fde15c414bdf867471d2dfea7b0f (127.0.98.3:42711): tablet 2225320e8a6e42bd9338dd8ca82a97b6 (table test-table [id=3fe4082e441e43d09536f72b6a686050]) successfully deleted
I20260927 00:46:54.958520 1491 catalog_manager.cc:4841] TS c37e48fbfc574f0e8ab332020c338011 (127.0.98.2:46389): tablet 2225320e8a6e42bd9338dd8ca82a97b6 (table test-table [id=3fe4082e441e43d09536f72b6a686050]) successfully deleted
I20260927 00:46:54.959448 1492 catalog_manager.cc:4841] TS 9b56cc27420a49a3b5569894cbbe9b6e (127.0.98.1:40385): tablet 2225320e8a6e42bd9338dd8ca82a97b6 (table test-table [id=3fe4082e441e43d09536f72b6a686050]) successfully deleted
Sep 27 00:46:54 dist-test-slave-86rs krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470014, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:46:54.970717 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.952834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36138 (local address 127.0.98.1:40385)
0927 00:46:54.952982 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:54.952987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:54.957264 (+ 4277us) server_negotiation.cc:407] Connection header received
0927 00:46:54.957579 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:54.957584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:54.957659 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:54.957728 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:54.959409 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.960271 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.961356 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.961716 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.961850 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.965917 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:54.965932 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:54.965956 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:54.966000 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:54.968439 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:54.969014 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:54.969022 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:54.969025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:54.969148 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:54.969438 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:54.969441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:54.969443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:54.969833 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:54.970067 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:54.970355 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 00:46:54.970504 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 00:46:54.971701 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36138
I20260927 00:46:54.984050 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:54.974274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49118 (local address 127.0.98.62:44639)
0927 00:46:54.974433 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:54.974438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:54.974561 (+ 123us) server_negotiation.cc:407] Connection header received
0927 00:46:54.974710 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:54.974714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:54.974777 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:54.974877 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:54.975875 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.976883 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.977860 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:54.978144 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:54.978281 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:54.980095 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:54.980100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:54.980103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:54.980205 (+ 102us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:54.982196 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:54.982680 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:54.982684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:54.982686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:54.982782 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:54.983066 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:54.983071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:54.983073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:54.983305 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:54.983456 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:54.983709 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 00:46:54.983845 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 00:46:54.984622 1503 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49118
I20260927 00:46:54.986049 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 1579
I20260927 00:46:54.995750 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 1719
I20260927 00:46:55.002854 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 1859
I20260927 00:46:55.011039 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 1475
2026-09-27T00:46:55Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3244 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:46:55 dist-test-slave-86rs krb5kdc[2079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:46:55 dist-test-slave-86rs krb5kdc[2079](info): set up 2 sockets
Sep 27 00:46:55 dist-test-slave-86rs krb5kdc[2079](info): commencing operation
krb5kdc: starting...
W20260927 00:46:56.153199 392 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:46:56 dist-test-slave-86rs krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:46:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:46:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:46:56.303805 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43249
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:56.450129 2092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:56.450573 2092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:56.450624 2092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:56.455695 2092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:46:56.455797 2092 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 00:46:56.455816 2092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:56.455838 2092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:56.455855 2092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:46:56.455873 2092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:46:56.462535 2092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43249
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:46:56.464130 2092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:56.465636 2092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:56.477818 2097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:56.477942 2098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:56.477942 2100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:56.478024 2092 server_base.cc:1034] running on GCE node
I20260927 00:46:56.479130 2092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:56.480966 2092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:56.482204 2092 hybrid_clock.cc:648] HybridClock initialized: now 1790470016482161 us; error 69 us; skew 500 ppm
Sep 27 00:46:56 dist-test-slave-86rs krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:56.486007 2092 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:46:56.487273 2092 webserver.cc:460] Webserver started at http://127.0.98.62:46235/ using document root <none> and password file <none>
I20260927 00:46:56.488036 2092 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:56.488107 2092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:56.488435 2092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:46:56.491274 2092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "7ef1885acaaa4265ab30b56cd692d00e"
format_stamp: "Formatted at 2026-09-27 00:46:56 on dist-test-slave-86rs"
server_key: "8cf9eba14c96aee540f9235d9126d2e4"
server_key_iv: "451be48230833af09eaff7b6ef87689f"
server_key_version: "encryptionkey@0"
I20260927 00:46:56.492031 2092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "7ef1885acaaa4265ab30b56cd692d00e"
format_stamp: "Formatted at 2026-09-27 00:46:56 on dist-test-slave-86rs"
server_key: "8cf9eba14c96aee540f9235d9126d2e4"
server_key_iv: "451be48230833af09eaff7b6ef87689f"
server_key_version: "encryptionkey@0"
I20260927 00:46:56.497540 2092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260927 00:46:56.500885 2107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:56.502673 2092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:46:56.502871 2092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "7ef1885acaaa4265ab30b56cd692d00e"
format_stamp: "Formatted at 2026-09-27 00:46:56 on dist-test-slave-86rs"
server_key: "8cf9eba14c96aee540f9235d9126d2e4"
server_key_iv: "451be48230833af09eaff7b6ef87689f"
server_key_version: "encryptionkey@0"
I20260927 00:46:56.503039 2092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:56.523139 2092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:56.527097 2092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:56.527367 2092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:56.537533 2092 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43249
I20260927 00:46:56.537547 2169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43249 every 8 connection(s)
I20260927 00:46:56.538820 2092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:46:56.542593 2170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:56.543542 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 2092
I20260927 00:46:56.543650 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:46:56.544003 392 external_mini_cluster.cc:1428] Setting key a6d3c18b66bc84cf6ad30977bb0cf8ce
I20260927 00:46:56.551258 2170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ef1885acaaa4265ab30b56cd692d00e: Bootstrap starting.
I20260927 00:46:56.554744 2170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ef1885acaaa4265ab30b56cd692d00e: Neither blocks nor log segments found. Creating new log.
Sep 27 00:46:56 dist-test-slave-86rs krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:56.555887 2170 log.cc:824] T 00000000000000000000000000000000 P 7ef1885acaaa4265ab30b56cd692d00e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:56.559173 2170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ef1885acaaa4265ab30b56cd692d00e: No bootstrap required, opened a new log
I20260927 00:46:56.563052 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:56.545981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.0.98.62:43249)
0927 00:46:56.546518 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:56.546555 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:56.546608 (+ 53us) server_negotiation.cc:407] Connection header received
0927 00:46:56.547545 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:56.547576 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:56.548127 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:56.548638 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:56.549454 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:56.551061 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:56.552130 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:56.552974 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:56.553162 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:56.556419 (+ 3257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:56.556427 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:56.556451 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:56.556493 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:56.559294 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:56.559785 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:56.559792 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:56.559799 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:56.559952 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:56.560220 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:56.560248 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:56.560249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:56.560848 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:56.561078 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:56.561344 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 00:46:56.561743 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":151,"threads_started":1}
I20260927 00:46:56.563944 2170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ef1885acaaa4265ab30b56cd692d00e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef1885acaaa4265ab30b56cd692d00e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43249 } }
I20260927 00:46:56.564291 2170 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ef1885acaaa4265ab30b56cd692d00e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:56.564379 2170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ef1885acaaa4265ab30b56cd692d00e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef1885acaaa4265ab30b56cd692d00e, State: Initialized, Role: FOLLOWER
I20260927 00:46:56.564922 2170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ef1885acaaa4265ab30b56cd692d00e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef1885acaaa4265ab30b56cd692d00e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43249 } }
I20260927 00:46:56.565102 2170 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ef1885acaaa4265ab30b56cd692d00e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:46:56.565193 2170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ef1885acaaa4265ab30b56cd692d00e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:46:56.565338 2170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ef1885acaaa4265ab30b56cd692d00e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:56.566730 2170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ef1885acaaa4265ab30b56cd692d00e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef1885acaaa4265ab30b56cd692d00e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43249 } }
I20260927 00:46:56.567198 2170 leader_election.cc:302] T 00000000000000000000000000000000 P 7ef1885acaaa4265ab30b56cd692d00e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ef1885acaaa4265ab30b56cd692d00e; no voters:
I20260927 00:46:56.567617 2170 leader_election.cc:288] T 00000000000000000000000000000000 P 7ef1885acaaa4265ab30b56cd692d00e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:46:56.567785 2175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ef1885acaaa4265ab30b56cd692d00e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:46:56.568099 2175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ef1885acaaa4265ab30b56cd692d00e [term 1 LEADER]: Becoming Leader. State: Replica: 7ef1885acaaa4265ab30b56cd692d00e, State: Running, Role: LEADER
I20260927 00:46:56.568550 2175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ef1885acaaa4265ab30b56cd692d00e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef1885acaaa4265ab30b56cd692d00e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43249 } }
I20260927 00:46:56.569355 2170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ef1885acaaa4265ab30b56cd692d00e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:46:56.570767 2177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ef1885acaaa4265ab30b56cd692d00e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ef1885acaaa4265ab30b56cd692d00e. Latest consensus state: current_term: 1 leader_uuid: "7ef1885acaaa4265ab30b56cd692d00e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef1885acaaa4265ab30b56cd692d00e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43249 } } }
I20260927 00:46:56.570930 2177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ef1885acaaa4265ab30b56cd692d00e [sys.catalog]: This master's current role is: LEADER
I20260927 00:46:56.571015 2176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ef1885acaaa4265ab30b56cd692d00e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ef1885acaaa4265ab30b56cd692d00e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef1885acaaa4265ab30b56cd692d00e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43249 } } }
I20260927 00:46:56.571117 2176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ef1885acaaa4265ab30b56cd692d00e [sys.catalog]: This master's current role is: LEADER
I20260927 00:46:56.571586 2184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:46:56.575680 2184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:46:56.583698 2184 catalog_manager.cc:1347] Generated new cluster ID: dc05a82b7b3145d08489ba1a1bc2c2bc
I20260927 00:46:56.583854 2184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:46:56.600198 2184 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:46:56.601632 2184 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:46:56.607630 2184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ef1885acaaa4265ab30b56cd692d00e: Generated new TSK 0
I20260927 00:46:56.608767 2184 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:46:56.660864 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43249
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:56.810703 2196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:56.811142 2196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:56.811203 2196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:56.816129 2196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:56.816300 2196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:56.816363 2196 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 00:46:56.816380 2196 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:46:56.816470 2196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:46:56.823783 2196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:46:56.825448 2196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:56.826874 2196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:56.840204 2201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:56.840171 2204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:56.840514 2196 server_base.cc:1034] running on GCE node
W20260927 00:46:56.840178 2202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:56.841275 2196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:56.842233 2196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:56.843467 2196 hybrid_clock.cc:648] HybridClock initialized: now 1790470016843402 us; error 64 us; skew 500 ppm
Sep 27 00:46:56 dist-test-slave-86rs krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:56.847255 2196 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:46:56.848647 2196 webserver.cc:460] Webserver started at http://127.0.98.1:32989/ using document root <none> and password file <none>
I20260927 00:46:56.849422 2196 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:56.849503 2196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:56.849740 2196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:46:56.852761 2196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-0/data/instance:
uuid: "c84e3dfe390e42589ab1c1d7731889fc"
format_stamp: "Formatted at 2026-09-27 00:46:56 on dist-test-slave-86rs"
server_key: "ab72f5732a11fa43ab29adc9cc036365"
server_key_iv: "cf0f41aceba08fe4746250daaef14f8c"
server_key_version: "encryptionkey@0"
I20260927 00:46:56.853560 2196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c84e3dfe390e42589ab1c1d7731889fc"
format_stamp: "Formatted at 2026-09-27 00:46:56 on dist-test-slave-86rs"
server_key: "ab72f5732a11fa43ab29adc9cc036365"
server_key_iv: "cf0f41aceba08fe4746250daaef14f8c"
server_key_version: "encryptionkey@0"
I20260927 00:46:56.859462 2196 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 00:46:56.863107 2211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:56.865054 2196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 00:46:56.865207 2196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "c84e3dfe390e42589ab1c1d7731889fc"
format_stamp: "Formatted at 2026-09-27 00:46:56 on dist-test-slave-86rs"
server_key: "ab72f5732a11fa43ab29adc9cc036365"
server_key_iv: "cf0f41aceba08fe4746250daaef14f8c"
server_key_version: "encryptionkey@0"
I20260927 00:46:56.865347 2196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:56.886183 2196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:56.890468 2196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:56.890799 2196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:56.892889 2196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:46:56.892951 2196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:56.893000 2196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:46:56.893016 2196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:46:56 dist-test-slave-86rs krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:56.906415 2196 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40547
I20260927 00:46:56.906458 2326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40547 every 8 connection(s)
I20260927 00:46:56.908104 2196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:46:56.910364 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:56.893951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55055 (local address 127.0.98.62:43249)
0927 00:46:56.894158 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:56.894165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:56.895359 (+ 1194us) server_negotiation.cc:407] Connection header received
0927 00:46:56.896843 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:56.896848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:56.896939 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:56.897061 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:56.898660 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:56.899283 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:56.900884 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:56.901182 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:56.901323 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:56.905343 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:56.905348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:56.905351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:56.905392 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:56.907495 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:56.908340 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:56.908349 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:56.908351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:56.908449 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:56.908926 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:56.908932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:56.908936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:56.909247 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:56.909380 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:56.909983 (+ 603us) server_negotiation.cc:299] Negotiation successful
0927 00:46:56.910166 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 00:46:56.911353 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 2196
I20260927 00:46:56.911504 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:46:56.911811 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:56.894550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43249 (local address 127.0.98.1:55055)
0927 00:46:56.895137 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:56.895200 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:56.896536 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:56.897359 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:56.897368 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:56.897810 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:56.898457 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:56.898473 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:56.899434 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:56.899439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:56.900737 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:56.900747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:56.901395 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:56.901400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:56.901672 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:56.902516 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:56.902551 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:56.905083 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:56.907676 (+ 2593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:56.907682 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:56.907708 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:56.908144 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:56.908566 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:56.908572 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:56.908574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:56.908799 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:56.909375 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:56.909384 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:56.909702 (+ 318us) client_negotiation.cc:769] Sending connection context
0927 00:46:56.909921 (+ 219us) client_negotiation.cc:240] Negotiation successful
0927 00:46:56.910368 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":174,"threads_started":1}
I20260927 00:46:56.911883 392 external_mini_cluster.cc:1428] Setting key 8158df59003bd069810387e3e629494f
I20260927 00:46:56.914028 2327 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43249
I20260927 00:46:56.914265 2327 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:56.915122 2327 heartbeater.cc:511] Master 127.0.98.62:43249 requested a full tablet report, sending...
I20260927 00:46:56.917593 2123 ts_manager.cc:194] Registered new tserver with Master: c84e3dfe390e42589ab1c1d7731889fc (127.0.98.1:40547)
I20260927 00:46:56.919008 2123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55055
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:46:56.928681 2334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:56.917208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43249 (local address 127.0.98.1:54307)
0927 00:46:56.919978 (+ 2770us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:56.919996 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:56.920147 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:56.920528 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:56.920532 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:56.920821 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:56.920961 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:56.920972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:56.922222 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:56.922226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:56.922841 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:56.922850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:56.923581 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:56.923585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:56.923673 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:56.924456 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:56.924479 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:56.924890 (+ 411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:56.926821 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:56.926824 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:56.926826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:56.927056 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:56.927428 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:56.927434 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:56.927439 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:56.927560 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:56.928041 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:56.928044 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:56.928146 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 00:46:56.928320 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 00:46:56.928487 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2653,"thread_start_us":122,"threads_started":1}
I20260927 00:46:56.928821 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:56.917307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54307 (local address 127.0.98.62:43249)
0927 00:46:56.917476 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:56.917482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:56.920084 (+ 2602us) server_negotiation.cc:407] Connection header received
0927 00:46:56.920301 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:56.920307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:56.920380 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:56.920485 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:56.921121 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:56.922039 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:56.922980 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:56.923402 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:56.923525 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:56.925063 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:56.925067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:56.925069 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:56.925095 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:56.926712 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:56.927207 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:56.927211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:56.927212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:56.927308 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:56.927647 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:56.927650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:56.927651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:56.927932 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:56.928047 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:56.928383 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 00:46:56.928565 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:46:56.947357 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43249
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:57.094300 2336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:57.094730 2336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:57.094794 2336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:57.099578 2336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:57.099710 2336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:57.099762 2336 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 00:46:57.099781 2336 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:46:57.099823 2336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:46:57.106586 2336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:46:57.108551 2336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:57.109977 2336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:57.123055 2344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:57.123044 2342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:57.123116 2341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:57.123509 2336 server_base.cc:1034] running on GCE node
I20260927 00:46:57.124213 2336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:57.125252 2336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:57.126441 2336 hybrid_clock.cc:648] HybridClock initialized: now 1790470017126395 us; error 65 us; skew 500 ppm
Sep 27 00:46:57 dist-test-slave-86rs krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:57.130671 2336 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:46:57.132153 2336 webserver.cc:460] Webserver started at http://127.0.98.2:34429/ using document root <none> and password file <none>
I20260927 00:46:57.133025 2336 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:57.133111 2336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:57.133409 2336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:46:57.136580 2336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-1/data/instance:
uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735"
format_stamp: "Formatted at 2026-09-27 00:46:57 on dist-test-slave-86rs"
server_key: "c4a938cdffe32c27dd708c849ea4243c"
server_key_iv: "379fc7750a8e753c02f74c346017b49c"
server_key_version: "encryptionkey@0"
I20260927 00:46:57.137430 2336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735"
format_stamp: "Formatted at 2026-09-27 00:46:57 on dist-test-slave-86rs"
server_key: "c4a938cdffe32c27dd708c849ea4243c"
server_key_iv: "379fc7750a8e753c02f74c346017b49c"
server_key_version: "encryptionkey@0"
I20260927 00:46:57.143026 2336 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 00:46:57.146744 2351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:57.148631 2336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:46:57.148826 2336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735"
format_stamp: "Formatted at 2026-09-27 00:46:57 on dist-test-slave-86rs"
server_key: "c4a938cdffe32c27dd708c849ea4243c"
server_key_iv: "379fc7750a8e753c02f74c346017b49c"
server_key_version: "encryptionkey@0"
I20260927 00:46:57.148999 2336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:57.170552 2336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:57.174786 2336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:57.175081 2336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:57.177418 2336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:46:57.177502 2336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:57.177585 2336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:46:57.177631 2336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:57.190841 2336 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43073
I20260927 00:46:57.190989 2466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43073 every 8 connection(s)
I20260927 00:46:57.192132 2336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 00:46:57 dist-test-slave-86rs krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:57.197093 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 2336
I20260927 00:46:57.197244 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:46:57.197664 392 external_mini_cluster.cc:1428] Setting key ee8312e7d5c9060df75aa6aeb48e0e16
I20260927 00:46:57.200891 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.178357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44881 (local address 127.0.98.62:43249)
0927 00:46:57.178537 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:57.178542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:57.179763 (+ 1221us) server_negotiation.cc:407] Connection header received
0927 00:46:57.181226 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:57.181231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:57.181317 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:57.181435 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:57.183126 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.183745 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.186049 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.186516 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.186635 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.194189 (+ 7554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:57.194197 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:57.194204 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:57.194257 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:57.197634 (+ 3377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:57.198608 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:57.198613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:57.198615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:57.198760 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:57.199301 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:57.199305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:57.199307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:57.199660 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:57.199781 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:57.200542 (+ 761us) server_negotiation.cc:299] Negotiation successful
0927 00:46:57.200700 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 00:46:57.202180 2360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.178989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43249 (local address 127.0.98.2:44881)
0927 00:46:57.179541 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:57.179613 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:57.180923 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:57.181724 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:57.181737 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:57.182232 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:57.182936 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:57.182954 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.183889 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.183894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:57.185833 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:57.185853 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.186966 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.186973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:57.187295 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.188790 (+ 1495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:57.188824 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:57.193831 (+ 5007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:57.197865 (+ 4034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:57.197876 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:57.197905 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:57.198479 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:57.198870 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:57.198873 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:57.198876 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:57.199143 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:57.199789 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:57.199800 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:57.200175 (+ 375us) client_negotiation.cc:769] Sending connection context
0927 00:46:57.200491 (+ 316us) client_negotiation.cc:240] Negotiation successful
0927 00:46:57.200971 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":165,"threads_started":1}
I20260927 00:46:57.203869 2467 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43249
I20260927 00:46:57.204101 2467 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:57.204864 2467 heartbeater.cc:511] Master 127.0.98.62:43249 requested a full tablet report, sending...
I20260927 00:46:57.206398 2122 ts_manager.cc:194] Registered new tserver with Master: 4d30b0c6d65d4dd1b23f9742e7ea1735 (127.0.98.2:43073)
I20260927 00:46:57.207355 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44881
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:46:57.217696 2474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.207337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43249 (local address 127.0.98.2:50737)
0927 00:46:57.207713 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:57.207732 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:57.207938 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:57.208444 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:57.208453 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:57.208751 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:57.208900 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:57.208912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.209965 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.209970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:57.210982 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:57.210993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.211475 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.211479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:57.211561 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.212411 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:57.212424 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:57.212829 (+ 405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:57.215147 (+ 2318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:57.215156 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:57.215162 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:57.215545 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:57.215879 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:57.215886 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:57.215892 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:57.216021 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:57.216950 (+ 929us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:57.216953 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:57.217097 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 00:46:57.217242 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 00:46:57.217460 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":197,"threads_started":1}
I20260927 00:46:57.217690 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.207199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50737 (local address 127.0.98.62:43249)
0927 00:46:57.207404 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:57.207411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:57.207817 (+ 406us) server_negotiation.cc:407] Connection header received
0927 00:46:57.208083 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:57.208087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:57.208145 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:57.208206 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:57.209053 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.209809 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.211103 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.211330 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.211433 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.212992 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:57.212995 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:57.212998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:57.213140 (+ 142us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:57.214991 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:57.215680 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:57.215683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:57.215685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:57.215774 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:57.216155 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:57.216158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:57.216159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:57.216705 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:57.216965 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:57.217310 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 00:46:57.217457 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:46:57.236559 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43249
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:46:57.380187 2476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:46:57.380662 2476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:46:57.380757 2476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:46:57.385442 2476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:46:57.385571 2476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:46:57.385651 2476 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 00:46:57.385691 2476 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:46:57.385766 2476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:46:57.392714 2476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:46:57.394624 2476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:46:57.396332 2476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:46:57.409294 2482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:46:57.409323 2481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:57.409495 2476 server_base.cc:1034] running on GCE node
W20260927 00:46:57.409296 2484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:46:57.410228 2476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:46:57.411096 2476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:46:57.412308 2476 hybrid_clock.cc:648] HybridClock initialized: now 1790470017412264 us; error 59 us; skew 500 ppm
Sep 27 00:46:57 dist-test-slave-86rs krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:46:57.415846 2476 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:46:57.417121 2476 webserver.cc:460] Webserver started at http://127.0.98.3:40219/ using document root <none> and password file <none>
I20260927 00:46:57.417883 2476 fs_manager.cc:359] Metadata directory not provided
I20260927 00:46:57.417984 2476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:46:57.418272 2476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:46:57.420989 2476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-2/data/instance:
uuid: "3dd218cc6b694cd485ad6e25b5b73547"
format_stamp: "Formatted at 2026-09-27 00:46:57 on dist-test-slave-86rs"
server_key: "f792d7ed78f1699192bb5ab76b232dc1"
server_key_iv: "20064a67ba6e93dd2ff712889b6ceda1"
server_key_version: "encryptionkey@0"
I20260927 00:46:57.421665 2476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3dd218cc6b694cd485ad6e25b5b73547"
format_stamp: "Formatted at 2026-09-27 00:46:57 on dist-test-slave-86rs"
server_key: "f792d7ed78f1699192bb5ab76b232dc1"
server_key_iv: "20064a67ba6e93dd2ff712889b6ceda1"
server_key_version: "encryptionkey@0"
I20260927 00:46:57.426432 2476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:46:57.429663 2491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:57.431339 2476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:46:57.431516 2476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "3dd218cc6b694cd485ad6e25b5b73547"
format_stamp: "Formatted at 2026-09-27 00:46:57 on dist-test-slave-86rs"
server_key: "f792d7ed78f1699192bb5ab76b232dc1"
server_key_iv: "20064a67ba6e93dd2ff712889b6ceda1"
server_key_version: "encryptionkey@0"
I20260927 00:46:57.431668 2476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:46:57.448436 2476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:46:57.452500 2476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:46:57.453051 2476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:46:57.455780 2476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:46:57.455900 2476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:46:57.455986 2476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:46:57.456022 2476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:46:57 dist-test-slave-86rs krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:57.469285 2476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40181
I20260927 00:46:57.469327 2606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40181 every 8 connection(s)
I20260927 00:46:57.470768 2476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:46:57.474310 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.457066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46811 (local address 127.0.98.62:43249)
0927 00:46:57.457257 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:57.457263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:57.458832 (+ 1569us) server_negotiation.cc:407] Connection header received
0927 00:46:57.460758 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:57.460763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:57.460854 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:57.460996 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:57.463445 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.464126 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.465702 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.465963 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.466099 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.469615 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:57.469620 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:57.469623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:57.469658 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:57.471809 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:57.472548 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:57.472551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:57.472553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:57.472646 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:57.473037 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:57.473040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:57.473041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:57.473290 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:57.473369 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:57.473996 (+ 627us) server_negotiation.cc:299] Negotiation successful
0927 00:46:57.474134 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 00:46:57.475560 2500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.457823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43249 (local address 127.0.98.3:46811)
0927 00:46:57.458450 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:57.458642 (+ 192us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:57.460409 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:57.461318 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:57.461335 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:57.462082 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:57.463170 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:57.463199 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.464313 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.464319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:57.465527 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:57.465539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.466150 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.466154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:57.466365 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.467191 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:57.467234 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:57.469381 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:57.471966 (+ 2585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:57.471971 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:57.471991 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:57.472417 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:57.472755 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:57.472758 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:57.472759 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:57.472937 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:57.473408 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:57.473416 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:57.473729 (+ 313us) client_negotiation.cc:769] Sending connection context
0927 00:46:57.473924 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 00:46:57.474329 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":182,"threads_started":1}
I20260927 00:46:57.476578 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 2476
I20260927 00:46:57.476761 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:46:57.477101 392 external_mini_cluster.cc:1428] Setting key ddb8fdc752db43bbb891709d410907eb
I20260927 00:46:57.477339 2607 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43249
I20260927 00:46:57.477516 2607 heartbeater.cc:464] Registering TS with master...
I20260927 00:46:57.478304 2607 heartbeater.cc:511] Master 127.0.98.62:43249 requested a full tablet report, sending...
I20260927 00:46:57.479575 2122 ts_manager.cc:194] Registered new tserver with Master: 3dd218cc6b694cd485ad6e25b5b73547 (127.0.98.3:40181)
I20260927 00:46:57.480352 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46811
I20260927 00:46:57.488301 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.480285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45933 (local address 127.0.98.62:43249)
0927 00:46:57.480412 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:57.480417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:57.480638 (+ 221us) server_negotiation.cc:407] Connection header received
0927 00:46:57.480761 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:57.480764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:57.480822 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:57.480885 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:57.481394 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.482155 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.482985 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.483217 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.483308 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.484867 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:57.484870 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:57.484873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:57.484898 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:57.486553 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:57.486994 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:57.486998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:57.486999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:57.487133 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:57.487427 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:57.487429 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:57.487430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:57.487634 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:57.487711 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:57.487947 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 00:46:57.488059 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 00:46:57.488310 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.480191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43249 (local address 127.0.98.3:45933)
0927 00:46:57.480526 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:57.480542 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:57.480658 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:57.480921 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:57.480925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:57.481158 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:57.481276 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:57.481283 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.482284 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.482286 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:57.482875 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:57.482882 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.483386 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.483389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:57.483506 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.484138 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:57.484150 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:57.484674 (+ 524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:57.486667 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:57.486669 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:57.486671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:57.486886 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:57.487250 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:57.487252 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:57.487254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:57.487340 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:57.487713 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:57.487716 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:57.487803 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 00:46:57.487886 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 00:46:57.488028 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":105,"threads_started":1}
I20260927 00:46:57.491801 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 00:46:57 dist-test-slave-86rs krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:46:57 dist-test-slave-86rs krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:57.522248 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.512664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54218 (local address 127.0.98.62:43249)
0927 00:46:57.512828 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:57.512831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:57.512846 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:46:57.512898 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:57.512900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:57.512946 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:57.513046 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:57.513881 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.514482 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.515412 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.515646 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.515763 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.518584 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:57.518588 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:57.518590 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:57.518626 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:57.520330 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:57.520825 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:57.520831 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:57.520837 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:57.520926 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:57.521231 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:57.521236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:57.521238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:57.521473 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:57.521626 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:57.521892 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 00:46:57.522032 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 00:46:57.526008 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:46:57.529820 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:46:57.548252 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.541086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40181 (local address 127.0.0.1:36662)
0927 00:46:57.542118 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:57.542133 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:57.542239 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:57.542819 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:57.542822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:57.542841 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:57.542963 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:57.542969 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.544538 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.544542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:57.546363 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:57.546373 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.547786 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.547790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:57.547895 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.547917 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:46:57.547981 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 00:46:57.548055 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"mutex_wait_us":1,"thread_start_us":123,"threads_started":1}
I20260927 00:46:57.548305 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.539977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40547 (local address 127.0.0.1:58764)
0927 00:46:57.541332 (+ 1355us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:57.541345 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:57.541437 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:57.543591 (+ 2154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:57.543594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:57.543614 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:57.543718 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:57.543724 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.545270 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.545274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:57.546426 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:57.546437 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.547568 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.547572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:57.547711 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.547757 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 00:46:57.547856 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 00:46:57.548009 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1296,"mutex_wait_us":44,"thread_start_us":88,"threads_started":1}
I20260927 00:46:57.548761 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.539968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43073 (local address 127.0.0.1:40260)
0927 00:46:57.540907 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:57.540959 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:57.541096 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:57.542274 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:57.542283 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:57.542310 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:57.542503 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:57.542510 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.545062 (+ 2552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.545065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:57.546442 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:57.546448 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.548338 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.548342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:57.548449 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.548476 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 00:46:57.548533 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 00:46:57.548619 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":75,"thread_start_us":114,"threads_started":1}
I20260927 00:46:57.549064 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.541486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36662 (local address 127.0.98.3:40181)
0927 00:46:57.541867 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:57.541878 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:57.542334 (+ 456us) server_negotiation.cc:407] Connection header received
0927 00:46:57.542423 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:57.542429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:57.542665 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:57.542810 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:57.543104 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.544409 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.546499 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.547404 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.547508 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.547727 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:57.548670 (+ 943us) server_negotiation.cc:299] Negotiation successful
0927 00:46:57.548811 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":129,"threads_started":1}
I20260927 00:46:57.549451 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.540480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40260 (local address 127.0.98.2:43073)
0927 00:46:57.541661 (+ 1181us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:57.541677 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:57.541706 (+ 29us) server_negotiation.cc:407] Connection header received
0927 00:46:57.541799 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:57.541808 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:57.542072 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:57.542684 (+ 612us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:57.542877 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.544930 (+ 2053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.547318 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.548172 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.548924 (+ 752us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.549135 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:57.549198 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 00:46:57.549298 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1025,"thread_start_us":136,"threads_started":1}
I20260927 00:46:57.549958 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.542057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58764 (local address 127.0.98.1:40547)
0927 00:46:57.543125 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:57.543132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:57.543168 (+ 36us) server_negotiation.cc:407] Connection header received
0927 00:46:57.543257 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:57.543262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:57.543448 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:57.543875 (+ 427us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:57.544049 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.545154 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.546572 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.547431 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.549290 (+ 1859us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.549439 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:57.549497 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 00:46:57.549814 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":956,"thread_start_us":121,"threads_started":1}
I20260927 00:46:57.552956 2401 tablet_service.cc:1467] Processing CreateTablet for tablet 82e7dd40d48a4592b81b5e78b3607b91 (DEFAULT_TABLE table=test-table [id=098052b285984de7b615247d7c3908e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:57.552959 2541 tablet_service.cc:1467] Processing CreateTablet for tablet 82e7dd40d48a4592b81b5e78b3607b91 (DEFAULT_TABLE table=test-table [id=098052b285984de7b615247d7c3908e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:57.554347 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82e7dd40d48a4592b81b5e78b3607b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:57.554363 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82e7dd40d48a4592b81b5e78b3607b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:57.554699 2261 tablet_service.cc:1467] Processing CreateTablet for tablet 82e7dd40d48a4592b81b5e78b3607b91 (DEFAULT_TABLE table=test-table [id=098052b285984de7b615247d7c3908e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:57.555971 2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82e7dd40d48a4592b81b5e78b3607b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:57.564105 2630 tablet_bootstrap.cc:492] T 82e7dd40d48a4592b81b5e78b3607b91 P c84e3dfe390e42589ab1c1d7731889fc: Bootstrap starting.
I20260927 00:46:57.564776 2631 tablet_bootstrap.cc:492] T 82e7dd40d48a4592b81b5e78b3607b91 P 4d30b0c6d65d4dd1b23f9742e7ea1735: Bootstrap starting.
I20260927 00:46:57.565965 2632 tablet_bootstrap.cc:492] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547: Bootstrap starting.
I20260927 00:46:57.567744 2631 tablet_bootstrap.cc:654] T 82e7dd40d48a4592b81b5e78b3607b91 P 4d30b0c6d65d4dd1b23f9742e7ea1735: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:57.567754 2630 tablet_bootstrap.cc:654] T 82e7dd40d48a4592b81b5e78b3607b91 P c84e3dfe390e42589ab1c1d7731889fc: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:57.568325 2632 tablet_bootstrap.cc:654] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:57.568940 2630 log.cc:824] T 82e7dd40d48a4592b81b5e78b3607b91 P c84e3dfe390e42589ab1c1d7731889fc: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:57.569223 2632 log.cc:824] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:57.569077 2631 log.cc:824] T 82e7dd40d48a4592b81b5e78b3607b91 P 4d30b0c6d65d4dd1b23f9742e7ea1735: Log is configured to *not* fsync() on all Append() calls
I20260927 00:46:57.571758 2632 tablet_bootstrap.cc:492] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547: No bootstrap required, opened a new log
I20260927 00:46:57.571758 2630 tablet_bootstrap.cc:492] T 82e7dd40d48a4592b81b5e78b3607b91 P c84e3dfe390e42589ab1c1d7731889fc: No bootstrap required, opened a new log
I20260927 00:46:57.572021 2630 ts_tablet_manager.cc:1397] T 82e7dd40d48a4592b81b5e78b3607b91 P c84e3dfe390e42589ab1c1d7731889fc: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260927 00:46:57.572021 2632 ts_tablet_manager.cc:1397] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260927 00:46:57.572096 2631 tablet_bootstrap.cc:492] T 82e7dd40d48a4592b81b5e78b3607b91 P 4d30b0c6d65d4dd1b23f9742e7ea1735: No bootstrap required, opened a new log
I20260927 00:46:57.572345 2631 ts_tablet_manager.cc:1397] T 82e7dd40d48a4592b81b5e78b3607b91 P 4d30b0c6d65d4dd1b23f9742e7ea1735: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 00:46:57.575788 2631 raft_consensus.cc:348] T 82e7dd40d48a4592b81b5e78b3607b91 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:57.576136 2631 raft_consensus.cc:374] T 82e7dd40d48a4592b81b5e78b3607b91 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:57.576215 2631 raft_consensus.cc:729] T 82e7dd40d48a4592b81b5e78b3607b91 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d30b0c6d65d4dd1b23f9742e7ea1735, State: Initialized, Role: FOLLOWER
I20260927 00:46:57.576687 2632 raft_consensus.cc:348] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:57.576895 2631 consensus_queue.cc:260] T 82e7dd40d48a4592b81b5e78b3607b91 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:57.577004 2632 raft_consensus.cc:374] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:57.577056 2632 raft_consensus.cc:729] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd218cc6b694cd485ad6e25b5b73547, State: Initialized, Role: FOLLOWER
I20260927 00:46:57.577155 2630 raft_consensus.cc:348] T 82e7dd40d48a4592b81b5e78b3607b91 P c84e3dfe390e42589ab1c1d7731889fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:57.577417 2630 raft_consensus.cc:374] T 82e7dd40d48a4592b81b5e78b3607b91 P c84e3dfe390e42589ab1c1d7731889fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:57.577517 2630 raft_consensus.cc:729] T 82e7dd40d48a4592b81b5e78b3607b91 P c84e3dfe390e42589ab1c1d7731889fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84e3dfe390e42589ab1c1d7731889fc, State: Initialized, Role: FOLLOWER
I20260927 00:46:57.577515 2632 consensus_queue.cc:260] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:57.577994 2630 consensus_queue.cc:260] T 82e7dd40d48a4592b81b5e78b3607b91 P c84e3dfe390e42589ab1c1d7731889fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:57.578091 2467 heartbeater.cc:503] Master 127.0.98.62:43249 was elected leader, sending a full tablet report...
I20260927 00:46:57.578667 2631 ts_tablet_manager.cc:1428] T 82e7dd40d48a4592b81b5e78b3607b91 P 4d30b0c6d65d4dd1b23f9742e7ea1735: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260927 00:46:57.579098 2630 ts_tablet_manager.cc:1428] T 82e7dd40d48a4592b81b5e78b3607b91 P c84e3dfe390e42589ab1c1d7731889fc: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260927 00:46:57.579403 2607 heartbeater.cc:503] Master 127.0.98.62:43249 was elected leader, sending a full tablet report...
I20260927 00:46:57.579880 2327 heartbeater.cc:503] Master 127.0.98.62:43249 was elected leader, sending a full tablet report...
I20260927 00:46:57.580178 2632 ts_tablet_manager.cc:1428] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260927 00:46:57.582770 2637 raft_consensus.cc:482] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:46:57.582940 2637 raft_consensus.cc:504] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:57.585333 2637 leader_election.cc:288] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d30b0c6d65d4dd1b23f9742e7ea1735 (127.0.98.2:43073), c84e3dfe390e42589ab1c1d7731889fc (127.0.98.1:40547)
I20260927 00:46:57.589078 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.585388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52043 (local address 127.0.98.1:40547)
0927 00:46:57.585526 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:57.585530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:57.585543 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:46:57.585596 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:57.585598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:57.585641 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:57.585734 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:57.586052 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.586801 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.587882 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.588454 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.588581 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.588664 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:57.588835 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 00:46:57.588927 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:46:57.589078 2500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.585100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40547 (local address 127.0.98.3:52043)
0927 00:46:57.585249 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:57.585268 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:57.585365 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:57.585748 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:57.585751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:57.585768 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:57.585918 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:57.585926 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.586898 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.586899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:57.587758 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:57.587766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.588584 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.588588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:57.588696 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.588720 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:46:57.588819 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 00:46:57.588914 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 00:46:57.590533 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.585104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43073 (local address 127.0.98.3:45713)
0927 00:46:57.586323 (+ 1219us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:57.586336 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:57.586420 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:57.586641 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:57.586643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:57.586652 (+ 9us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:57.586751 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:57.586754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.587535 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.587545 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:57.588921 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:57.588931 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.590063 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.590067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:57.590166 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.590194 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 00:46:57.590279 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 00:46:57.590364 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1137,"mutex_wait_us":716,"thread_start_us":254,"threads_started":1}
I20260927 00:46:57.590517 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.585392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45713 (local address 127.0.98.2:43073)
0927 00:46:57.585539 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:57.585544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:57.586429 (+ 885us) server_negotiation.cc:407] Connection header received
0927 00:46:57.586493 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:57.586497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:57.586551 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:57.586638 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:57.586836 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.587414 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.589073 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.589630 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.589704 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.589789 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:57.590313 (+ 524us) server_negotiation.cc:299] Negotiation successful
0927 00:46:57.590401 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 00:46:57.590471 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e7dd40d48a4592b81b5e78b3607b91" candidate_uuid: "3dd218cc6b694cd485ad6e25b5b73547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84e3dfe390e42589ab1c1d7731889fc" is_pre_election: true
I20260927 00:46:57.590946 2281 raft_consensus.cc:2454] T 82e7dd40d48a4592b81b5e78b3607b91 P c84e3dfe390e42589ab1c1d7731889fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dd218cc6b694cd485ad6e25b5b73547 in term 0.
I20260927 00:46:57.591171 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e7dd40d48a4592b81b5e78b3607b91" candidate_uuid: "3dd218cc6b694cd485ad6e25b5b73547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" is_pre_election: true
I20260927 00:46:57.591503 2421 raft_consensus.cc:2454] T 82e7dd40d48a4592b81b5e78b3607b91 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dd218cc6b694cd485ad6e25b5b73547 in term 0.
I20260927 00:46:57.591568 2495 leader_election.cc:302] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dd218cc6b694cd485ad6e25b5b73547, c84e3dfe390e42589ab1c1d7731889fc; no voters:
I20260927 00:46:57.591873 2637 raft_consensus.cc:2781] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:46:57.591965 2637 raft_consensus.cc:482] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:46:57.592029 2637 raft_consensus.cc:3037] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:57.593879 2637 raft_consensus.cc:504] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:57.594446 2637 leader_election.cc:288] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547 [CANDIDATE]: Term 1 election: Requested vote from peers 4d30b0c6d65d4dd1b23f9742e7ea1735 (127.0.98.2:43073), c84e3dfe390e42589ab1c1d7731889fc (127.0.98.1:40547)
I20260927 00:46:57.594894 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e7dd40d48a4592b81b5e78b3607b91" candidate_uuid: "3dd218cc6b694cd485ad6e25b5b73547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84e3dfe390e42589ab1c1d7731889fc"
I20260927 00:46:57.595016 2281 raft_consensus.cc:3037] T 82e7dd40d48a4592b81b5e78b3607b91 P c84e3dfe390e42589ab1c1d7731889fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:57.596196 2281 raft_consensus.cc:2454] T 82e7dd40d48a4592b81b5e78b3607b91 P c84e3dfe390e42589ab1c1d7731889fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dd218cc6b694cd485ad6e25b5b73547 in term 1.
I20260927 00:46:57.596647 2495 leader_election.cc:302] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dd218cc6b694cd485ad6e25b5b73547, c84e3dfe390e42589ab1c1d7731889fc; no voters:
I20260927 00:46:57.596906 2637 raft_consensus.cc:2781] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:46:57.597200 2637 raft_consensus.cc:686] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547 [term 1 LEADER]: Becoming Leader. State: Replica: 3dd218cc6b694cd485ad6e25b5b73547, State: Running, Role: LEADER
I20260927 00:46:57.597472 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e7dd40d48a4592b81b5e78b3607b91" candidate_uuid: "3dd218cc6b694cd485ad6e25b5b73547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735"
I20260927 00:46:57.597661 2421 raft_consensus.cc:3037] T 82e7dd40d48a4592b81b5e78b3607b91 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:57.597689 2637 consensus_queue.cc:237] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:57.599182 2421 raft_consensus.cc:2454] T 82e7dd40d48a4592b81b5e78b3607b91 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dd218cc6b694cd485ad6e25b5b73547 in term 1.
I20260927 00:46:57.602160 2122 catalog_manager.cc:5495] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3dd218cc6b694cd485ad6e25b5b73547 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3dd218cc6b694cd485ad6e25b5b73547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } health_report { overall_health: HEALTHY } } }
Sep 27 00:46:57 dist-test-slave-86rs krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
W20260927 00:46:57.659920 2328 tablet.cc:2369] T 82e7dd40d48a4592b81b5e78b3607b91 P c84e3dfe390e42589ab1c1d7731889fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:46:57.662240 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.651073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54228 (local address 127.0.98.62:43249)
0927 00:46:57.651301 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:57.651306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:57.651482 (+ 176us) server_negotiation.cc:407] Connection header received
0927 00:46:57.651695 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:57.651698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:57.651749 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:57.651843 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:57.652791 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.653366 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.654312 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.654535 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.654669 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.658147 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:57.658153 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:57.658156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:57.658189 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:57.659932 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:57.660646 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:57.660649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:57.660651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:57.660738 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:57.661088 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:57.661091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:57.661092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:57.661406 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:57.661497 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:57.661874 (+ 377us) server_negotiation.cc:299] Negotiation successful
0927 00:46:57.662041 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":113}
I20260927 00:46:57.662251 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.650831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43249 (local address 127.0.0.1:54228)
0927 00:46:57.651371 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:57.651385 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:57.651562 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:57.651862 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:57.651865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:57.652541 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:46:57.652661 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:57.652670 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.653512 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.653515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:57.654193 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:57.654204 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.654689 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.654692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:57.654801 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.655759 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:46:57.655775 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:46:57.657893 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:46:57.660106 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:57.660120 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:57.660129 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:57.660516 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:57.660843 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:46:57.660846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:46:57.660848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:46:57.660953 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:46:57.661545 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:46:57.661552 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:46:57.661687 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 00:46:57.661802 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 00:46:57.662028 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":1,"thread_start_us":185,"threads_started":1}
I20260927 00:46:57.668005 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54228:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 00:46:57.669245 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:46:57.675971 2541 tablet_service.cc:1467] Processing CreateTablet for tablet dfe2aa18dee4454b9ba7756ea2ae594b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1505f565a22a452cad60892fe3bf65ab]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:46:57.675881 2261 tablet_service.cc:1467] Processing CreateTablet for tablet dfe2aa18dee4454b9ba7756ea2ae594b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1505f565a22a452cad60892fe3bf65ab]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:46:57.675881 2401 tablet_service.cc:1467] Processing CreateTablet for tablet dfe2aa18dee4454b9ba7756ea2ae594b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1505f565a22a452cad60892fe3bf65ab]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:46:57.676891 2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe2aa18dee4454b9ba7756ea2ae594b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:57.676891 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe2aa18dee4454b9ba7756ea2ae594b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:57.676892 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe2aa18dee4454b9ba7756ea2ae594b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:57.680276 2631 tablet_bootstrap.cc:492] T dfe2aa18dee4454b9ba7756ea2ae594b P 4d30b0c6d65d4dd1b23f9742e7ea1735: Bootstrap starting.
I20260927 00:46:57.680348 2630 tablet_bootstrap.cc:492] T dfe2aa18dee4454b9ba7756ea2ae594b P c84e3dfe390e42589ab1c1d7731889fc: Bootstrap starting.
I20260927 00:46:57.680728 2632 tablet_bootstrap.cc:492] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547: Bootstrap starting.
I20260927 00:46:57.681406 2631 tablet_bootstrap.cc:654] T dfe2aa18dee4454b9ba7756ea2ae594b P 4d30b0c6d65d4dd1b23f9742e7ea1735: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:57.681535 2630 tablet_bootstrap.cc:654] T dfe2aa18dee4454b9ba7756ea2ae594b P c84e3dfe390e42589ab1c1d7731889fc: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:57.681897 2632 tablet_bootstrap.cc:654] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:57.683127 2630 tablet_bootstrap.cc:492] T dfe2aa18dee4454b9ba7756ea2ae594b P c84e3dfe390e42589ab1c1d7731889fc: No bootstrap required, opened a new log
I20260927 00:46:57.683244 2630 ts_tablet_manager.cc:1397] T dfe2aa18dee4454b9ba7756ea2ae594b P c84e3dfe390e42589ab1c1d7731889fc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:46:57.683306 2631 tablet_bootstrap.cc:492] T dfe2aa18dee4454b9ba7756ea2ae594b P 4d30b0c6d65d4dd1b23f9742e7ea1735: No bootstrap required, opened a new log
I20260927 00:46:57.683398 2631 ts_tablet_manager.cc:1397] T dfe2aa18dee4454b9ba7756ea2ae594b P 4d30b0c6d65d4dd1b23f9742e7ea1735: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:46:57.683507 2632 tablet_bootstrap.cc:492] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547: No bootstrap required, opened a new log
I20260927 00:46:57.683612 2632 ts_tablet_manager.cc:1397] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:46:57.683858 2631 raft_consensus.cc:348] T dfe2aa18dee4454b9ba7756ea2ae594b P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:57.683857 2630 raft_consensus.cc:348] T dfe2aa18dee4454b9ba7756ea2ae594b P c84e3dfe390e42589ab1c1d7731889fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:57.684008 2631 raft_consensus.cc:374] T dfe2aa18dee4454b9ba7756ea2ae594b P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:57.684008 2630 raft_consensus.cc:374] T dfe2aa18dee4454b9ba7756ea2ae594b P c84e3dfe390e42589ab1c1d7731889fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:57.684044 2630 raft_consensus.cc:729] T dfe2aa18dee4454b9ba7756ea2ae594b P c84e3dfe390e42589ab1c1d7731889fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84e3dfe390e42589ab1c1d7731889fc, State: Initialized, Role: FOLLOWER
I20260927 00:46:57.684044 2631 raft_consensus.cc:729] T dfe2aa18dee4454b9ba7756ea2ae594b P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d30b0c6d65d4dd1b23f9742e7ea1735, State: Initialized, Role: FOLLOWER
I20260927 00:46:57.684103 2632 raft_consensus.cc:348] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:57.684264 2632 raft_consensus.cc:374] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:57.684202 2630 consensus_queue.cc:260] T dfe2aa18dee4454b9ba7756ea2ae594b P c84e3dfe390e42589ab1c1d7731889fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:57.684314 2632 raft_consensus.cc:729] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd218cc6b694cd485ad6e25b5b73547, State: Initialized, Role: FOLLOWER
I20260927 00:46:57.684259 2631 consensus_queue.cc:260] T dfe2aa18dee4454b9ba7756ea2ae594b P 4d30b0c6d65d4dd1b23f9742e7ea1735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:57.684453 2632 consensus_queue.cc:260] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:57.684545 2630 ts_tablet_manager.cc:1428] T dfe2aa18dee4454b9ba7756ea2ae594b P c84e3dfe390e42589ab1c1d7731889fc: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260927 00:46:57.684571 2636 tablet_replica.cc:439] T dfe2aa18dee4454b9ba7756ea2ae594b P 4d30b0c6d65d4dd1b23f9742e7ea1735: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } } }
I20260927 00:46:57.684732 2632 ts_tablet_manager.cc:1428] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:46:57.684741 2636 tablet_replica.cc:442] T dfe2aa18dee4454b9ba7756ea2ae594b P 4d30b0c6d65d4dd1b23f9742e7ea1735: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:46:57.684985 2631 ts_tablet_manager.cc:1428] T dfe2aa18dee4454b9ba7756ea2ae594b P 4d30b0c6d65d4dd1b23f9742e7ea1735: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 00:46:57.684964 2638 tablet_replica.cc:439] T dfe2aa18dee4454b9ba7756ea2ae594b P c84e3dfe390e42589ab1c1d7731889fc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } } }
I20260927 00:46:57.685114 2638 tablet_replica.cc:442] T dfe2aa18dee4454b9ba7756ea2ae594b P c84e3dfe390e42589ab1c1d7731889fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:46:57.685113 2637 tablet_replica.cc:439] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } } }
I20260927 00:46:57.685240 2637 tablet_replica.cc:442] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 00:46:57.695246 2468 tablet.cc:2369] T 82e7dd40d48a4592b81b5e78b3607b91 P 4d30b0c6d65d4dd1b23f9742e7ea1735: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:46:57.722095 2608 tablet.cc:2369] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:46:57.777894 2637 raft_consensus.cc:482] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:46:57.778131 2637 raft_consensus.cc:504] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:57.778784 2637 leader_election.cc:288] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c84e3dfe390e42589ab1c1d7731889fc (127.0.98.1:40547), 4d30b0c6d65d4dd1b23f9742e7ea1735 (127.0.98.2:43073)
I20260927 00:46:57.779403 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe2aa18dee4454b9ba7756ea2ae594b" candidate_uuid: "3dd218cc6b694cd485ad6e25b5b73547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84e3dfe390e42589ab1c1d7731889fc" is_pre_election: true
I20260927 00:46:57.779575 2281 raft_consensus.cc:2454] T dfe2aa18dee4454b9ba7756ea2ae594b P c84e3dfe390e42589ab1c1d7731889fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dd218cc6b694cd485ad6e25b5b73547 in term 0.
I20260927 00:46:57.779562 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe2aa18dee4454b9ba7756ea2ae594b" candidate_uuid: "3dd218cc6b694cd485ad6e25b5b73547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" is_pre_election: true
I20260927 00:46:57.779786 2421 raft_consensus.cc:2454] T dfe2aa18dee4454b9ba7756ea2ae594b P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dd218cc6b694cd485ad6e25b5b73547 in term 0.
I20260927 00:46:57.780130 2495 leader_election.cc:302] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dd218cc6b694cd485ad6e25b5b73547, c84e3dfe390e42589ab1c1d7731889fc; no voters:
I20260927 00:46:57.780413 2637 raft_consensus.cc:2781] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:46:57.780514 2637 raft_consensus.cc:482] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:46:57.780562 2637 raft_consensus.cc:3037] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:57.781936 2637 raft_consensus.cc:504] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:57.782389 2637 leader_election.cc:288] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547 [CANDIDATE]: Term 1 election: Requested vote from peers c84e3dfe390e42589ab1c1d7731889fc (127.0.98.1:40547), 4d30b0c6d65d4dd1b23f9742e7ea1735 (127.0.98.2:43073)
I20260927 00:46:57.782897 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe2aa18dee4454b9ba7756ea2ae594b" candidate_uuid: "3dd218cc6b694cd485ad6e25b5b73547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735"
I20260927 00:46:57.782898 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe2aa18dee4454b9ba7756ea2ae594b" candidate_uuid: "3dd218cc6b694cd485ad6e25b5b73547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84e3dfe390e42589ab1c1d7731889fc"
I20260927 00:46:57.783057 2421 raft_consensus.cc:3037] T dfe2aa18dee4454b9ba7756ea2ae594b P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:57.783062 2281 raft_consensus.cc:3037] T dfe2aa18dee4454b9ba7756ea2ae594b P c84e3dfe390e42589ab1c1d7731889fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:57.784593 2281 raft_consensus.cc:2454] T dfe2aa18dee4454b9ba7756ea2ae594b P c84e3dfe390e42589ab1c1d7731889fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dd218cc6b694cd485ad6e25b5b73547 in term 1.
I20260927 00:46:57.784590 2421 raft_consensus.cc:2454] T dfe2aa18dee4454b9ba7756ea2ae594b P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dd218cc6b694cd485ad6e25b5b73547 in term 1.
I20260927 00:46:57.784981 2494 leader_election.cc:302] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3dd218cc6b694cd485ad6e25b5b73547, 4d30b0c6d65d4dd1b23f9742e7ea1735, c84e3dfe390e42589ab1c1d7731889fc; no voters:
I20260927 00:46:57.785293 2637 raft_consensus.cc:2781] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:46:57.785409 2637 raft_consensus.cc:686] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547 [term 1 LEADER]: Becoming Leader. State: Replica: 3dd218cc6b694cd485ad6e25b5b73547, State: Running, Role: LEADER
I20260927 00:46:57.785691 2637 consensus_queue.cc:237] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:57.787034 2640 tablet_replica.cc:439] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547: TxnStatusTablet state changed. Reason: New leader 3dd218cc6b694cd485ad6e25b5b73547. Latest consensus state: current_term: 1 leader_uuid: "3dd218cc6b694cd485ad6e25b5b73547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } } }
I20260927 00:46:57.787189 2640 tablet_replica.cc:442] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:46:57.787585 2652 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 00:46:57.787912 2652 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 00:46:57.788558 2123 catalog_manager.cc:5495] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3dd218cc6b694cd485ad6e25b5b73547 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3dd218cc6b694cd485ad6e25b5b73547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } health_report { overall_health: HEALTHY } } }
I20260927 00:46:57.821625 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.817247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36672 (local address 127.0.98.3:40181)
0927 00:46:57.817421 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:57.817427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:57.817533 (+ 106us) server_negotiation.cc:407] Connection header received
0927 00:46:57.817819 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:57.817822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:57.817889 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:57.818000 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:46:57.818284 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.818835 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.819799 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.820030 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.820149 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.820525 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:46:57.821136 (+ 611us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:46:57.821296 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:57.821361 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 00:46:57.821433 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 00:46:57.821625 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.817119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40181 (local address 127.0.0.1:36672)
0927 00:46:57.817444 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:57.817460 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:57.817643 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:57.818010 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:57.818013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:57.818020 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:46:57.818163 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:57.818170 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.818979 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.818983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:57.819665 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:57.819674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.820153 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.820157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:57.820295 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.820351 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:46:57.821255 (+ 904us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:46:57.821275 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 00:46:57.821345 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 00:46:57.821466 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201}
I20260927 00:46:57.829313 2281 raft_consensus.cc:1261] T dfe2aa18dee4454b9ba7756ea2ae594b P c84e3dfe390e42589ab1c1d7731889fc [term 1 FOLLOWER]: Refusing update from remote peer 3dd218cc6b694cd485ad6e25b5b73547: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:46:57.829313 2421 raft_consensus.cc:1261] T dfe2aa18dee4454b9ba7756ea2ae594b P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 1 FOLLOWER]: Refusing update from remote peer 3dd218cc6b694cd485ad6e25b5b73547: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:46:57.830197 2637 consensus_queue.cc:1035] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:46:57.830466 2640 consensus_queue.cc:1035] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:46:57.835559 2636 tablet_replica.cc:439] T dfe2aa18dee4454b9ba7756ea2ae594b P 4d30b0c6d65d4dd1b23f9742e7ea1735: TxnStatusTablet state changed. Reason: New leader 3dd218cc6b694cd485ad6e25b5b73547. Latest consensus state: current_term: 1 leader_uuid: "3dd218cc6b694cd485ad6e25b5b73547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } } }
I20260927 00:46:57.835585 2638 tablet_replica.cc:439] T dfe2aa18dee4454b9ba7756ea2ae594b P c84e3dfe390e42589ab1c1d7731889fc: TxnStatusTablet state changed. Reason: New leader 3dd218cc6b694cd485ad6e25b5b73547. Latest consensus state: current_term: 1 leader_uuid: "3dd218cc6b694cd485ad6e25b5b73547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } } }
I20260927 00:46:57.835739 2636 tablet_replica.cc:442] T dfe2aa18dee4454b9ba7756ea2ae594b P 4d30b0c6d65d4dd1b23f9742e7ea1735: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:46:57.835741 2638 tablet_replica.cc:442] T dfe2aa18dee4454b9ba7756ea2ae594b P c84e3dfe390e42589ab1c1d7731889fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:46:57.840083 2636 tablet_replica.cc:439] T dfe2aa18dee4454b9ba7756ea2ae594b P 4d30b0c6d65d4dd1b23f9742e7ea1735: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3dd218cc6b694cd485ad6e25b5b73547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } } }
I20260927 00:46:57.840301 2636 tablet_replica.cc:442] T dfe2aa18dee4454b9ba7756ea2ae594b P 4d30b0c6d65d4dd1b23f9742e7ea1735: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:46:57.840348 2640 tablet_replica.cc:439] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3dd218cc6b694cd485ad6e25b5b73547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } } }
I20260927 00:46:57.840474 2640 tablet_replica.cc:442] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:46:57.841259 2655 mvcc.cc:205] Tried to move back new op lower bound from 7333765193019023360 to 7333765192852062208. Current Snapshot: MvccSnapshot[applied={T|T < 7333765193019023360}]
I20260927 00:46:57.842823 2657 mvcc.cc:205] Tried to move back new op lower bound from 7333765193019023360 to 7333765192852062208. Current Snapshot: MvccSnapshot[applied={T|T < 7333765193019023360}]
I20260927 00:46:57.844491 2637 tablet_replica.cc:439] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3dd218cc6b694cd485ad6e25b5b73547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } } }
I20260927 00:46:57.844633 2637 tablet_replica.cc:442] T dfe2aa18dee4454b9ba7756ea2ae594b P 3dd218cc6b694cd485ad6e25b5b73547: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:46:57.846863 2656 mvcc.cc:205] Tried to move back new op lower bound from 7333765193019023360 to 7333765192852062208. Current Snapshot: MvccSnapshot[applied={T|T < 7333765193019023360}]
I20260927 00:46:57.847388 2638 tablet_replica.cc:439] T dfe2aa18dee4454b9ba7756ea2ae594b P c84e3dfe390e42589ab1c1d7731889fc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3dd218cc6b694cd485ad6e25b5b73547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } } }
I20260927 00:46:57.847541 2638 tablet_replica.cc:442] T dfe2aa18dee4454b9ba7756ea2ae594b P c84e3dfe390e42589ab1c1d7731889fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:46:57.855266 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.850954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36678 (local address 127.0.98.3:40181)
0927 00:46:57.851307 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:57.851310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:57.851321 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:46:57.851363 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:57.851365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:57.851407 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:57.851494 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:46:57.851782 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.852411 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.853813 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.854233 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.854347 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.854656 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:46:57.854808 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:46:57.854901 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:57.855029 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 00:46:57.855094 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278}
I20260927 00:46:57.870962 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.866748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34447 (local address 127.0.98.3:40181)
0927 00:46:57.867132 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:57.867135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:57.867145 (+ 10us) server_negotiation.cc:407] Connection header received
0927 00:46:57.867177 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:57.867179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:57.867216 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:57.867305 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:46:57.867660 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.868250 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.869399 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.869702 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.869885 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.870278 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:46:57.870399 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:46:57.870483 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:57.870636 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 00:46:57.870740 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":323}
I20260927 00:46:57.870973 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:57.866679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40181 (local address 127.0.98.3:34447)
0927 00:46:57.866890 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:57.866905 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:57.867008 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:57.867312 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:57.867315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:57.867321 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:46:57.867544 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:57.867551 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.868394 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.868400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:57.869242 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:57.869255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:57.869826 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:57.869830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:57.869944 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:57.870020 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:46:57.870527 (+ 507us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:46:57.870536 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 00:46:57.870604 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 00:46:57.870743 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":36}
I20260927 00:46:57.879664 2280 raft_consensus.cc:1261] T 82e7dd40d48a4592b81b5e78b3607b91 P c84e3dfe390e42589ab1c1d7731889fc [term 1 FOLLOWER]: Refusing update from remote peer 3dd218cc6b694cd485ad6e25b5b73547: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:46:57.879675 2421 raft_consensus.cc:1261] T 82e7dd40d48a4592b81b5e78b3607b91 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 1 FOLLOWER]: Refusing update from remote peer 3dd218cc6b694cd485ad6e25b5b73547: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:46:57.880115 2637 consensus_queue.cc:1035] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:46:57.880352 2640 consensus_queue.cc:1035] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:46:57.930986 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:46:57.931206 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:46:57.934453 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ef1885acaaa4265ab30b56cd692d00e: Sending DeleteTablet for 3 replicas of tablet 82e7dd40d48a4592b81b5e78b3607b91
I20260927 00:46:57.935508 2261 tablet_service.cc:1514] Processing DeleteTablet for tablet 82e7dd40d48a4592b81b5e78b3607b91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:46:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58764
I20260927 00:46:57.935562 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet 82e7dd40d48a4592b81b5e78b3607b91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:46:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40260
I20260927 00:46:57.935731 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 82e7dd40d48a4592b81b5e78b3607b91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:46:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36662
I20260927 00:46:57.936272 2673 tablet_replica.cc:330] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547: stopping tablet replica
I20260927 00:46:57.936780 2673 raft_consensus.cc:2229] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:46:57.936843 2672 tablet_replica.cc:330] T 82e7dd40d48a4592b81b5e78b3607b91 P 4d30b0c6d65d4dd1b23f9742e7ea1735: stopping tablet replica
I20260927 00:46:57.937219 2672 raft_consensus.cc:2229] T 82e7dd40d48a4592b81b5e78b3607b91 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:46:57.937156 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 00:46:57.937295 2673 raft_consensus.cc:2258] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:46:57.937462 2672 raft_consensus.cc:2258] T 82e7dd40d48a4592b81b5e78b3607b91 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 00:46:57.937803 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:46:57.939263 2672 ts_tablet_manager.cc:1905] T 82e7dd40d48a4592b81b5e78b3607b91 P 4d30b0c6d65d4dd1b23f9742e7ea1735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:46:57.940378 2673 ts_tablet_manager.cc:1905] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:46:57.943293 2672 ts_tablet_manager.cc:1918] T 82e7dd40d48a4592b81b5e78b3607b91 P 4d30b0c6d65d4dd1b23f9742e7ea1735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:46:57.943445 2672 log.cc:1196] T 82e7dd40d48a4592b81b5e78b3607b91 P 4d30b0c6d65d4dd1b23f9742e7ea1735: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-1/wal/wals/82e7dd40d48a4592b81b5e78b3607b91
I20260927 00:46:57.943743 2671 tablet_replica.cc:330] T 82e7dd40d48a4592b81b5e78b3607b91 P c84e3dfe390e42589ab1c1d7731889fc: stopping tablet replica
I20260927 00:46:57.943820 2672 ts_tablet_manager.cc:1939] T 82e7dd40d48a4592b81b5e78b3607b91 P 4d30b0c6d65d4dd1b23f9742e7ea1735: Deleting consensus metadata
I20260927 00:46:57.944376 2671 raft_consensus.cc:2229] T 82e7dd40d48a4592b81b5e78b3607b91 P c84e3dfe390e42589ab1c1d7731889fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:46:57.944751 2671 raft_consensus.cc:2258] T 82e7dd40d48a4592b81b5e78b3607b91 P c84e3dfe390e42589ab1c1d7731889fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:46:57.945016 2401 tablet_service.cc:1467] Processing CreateTablet for tablet 6af7137078354672a09a98268e1953b5 (DEFAULT_TABLE table=test-table [id=97c4dd1feb924fa29e4cf14197b3c392]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:57.945238 2261 tablet_service.cc:1467] Processing CreateTablet for tablet 6af7137078354672a09a98268e1953b5 (DEFAULT_TABLE table=test-table [id=97c4dd1feb924fa29e4cf14197b3c392]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:57.945371 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af7137078354672a09a98268e1953b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:57.945542 2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af7137078354672a09a98268e1953b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:57.946271 2673 ts_tablet_manager.cc:1918] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:46:57.946444 2673 log.cc:1196] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-2/wal/wals/82e7dd40d48a4592b81b5e78b3607b91
I20260927 00:46:57.946851 2671 ts_tablet_manager.cc:1905] T 82e7dd40d48a4592b81b5e78b3607b91 P c84e3dfe390e42589ab1c1d7731889fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:46:57.946892 2673 ts_tablet_manager.cc:1939] T 82e7dd40d48a4592b81b5e78b3607b91 P 3dd218cc6b694cd485ad6e25b5b73547: Deleting consensus metadata
I20260927 00:46:57.948508 2630 tablet_bootstrap.cc:492] T 6af7137078354672a09a98268e1953b5 P c84e3dfe390e42589ab1c1d7731889fc: Bootstrap starting.
I20260927 00:46:57.948997 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 6af7137078354672a09a98268e1953b5 (DEFAULT_TABLE table=test-table [id=97c4dd1feb924fa29e4cf14197b3c392]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:57.949352 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af7137078354672a09a98268e1953b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:57.949636 2630 tablet_bootstrap.cc:654] T 6af7137078354672a09a98268e1953b5 P c84e3dfe390e42589ab1c1d7731889fc: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:57.950052 2671 ts_tablet_manager.cc:1918] T 82e7dd40d48a4592b81b5e78b3607b91 P c84e3dfe390e42589ab1c1d7731889fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:46:57.950148 2671 log.cc:1196] T 82e7dd40d48a4592b81b5e78b3607b91 P c84e3dfe390e42589ab1c1d7731889fc: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-0/wal/wals/82e7dd40d48a4592b81b5e78b3607b91
I20260927 00:46:57.950472 2631 tablet_bootstrap.cc:492] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735: Bootstrap starting.
I20260927 00:46:57.950496 2671 ts_tablet_manager.cc:1939] T 82e7dd40d48a4592b81b5e78b3607b91 P c84e3dfe390e42589ab1c1d7731889fc: Deleting consensus metadata
I20260927 00:46:57.951619 2631 tablet_bootstrap.cc:654] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:57.951905 2111 catalog_manager.cc:4841] TS c84e3dfe390e42589ab1c1d7731889fc (127.0.98.1:40547): tablet 82e7dd40d48a4592b81b5e78b3607b91 (table test-table [id=098052b285984de7b615247d7c3908e6]) successfully deleted
I20260927 00:46:57.952467 2110 catalog_manager.cc:4841] TS 4d30b0c6d65d4dd1b23f9742e7ea1735 (127.0.98.2:43073): tablet 82e7dd40d48a4592b81b5e78b3607b91 (table test-table [id=098052b285984de7b615247d7c3908e6]) successfully deleted
I20260927 00:46:57.952649 2110 catalog_manager.cc:4841] TS 3dd218cc6b694cd485ad6e25b5b73547 (127.0.98.3:40181): tablet 82e7dd40d48a4592b81b5e78b3607b91 (table test-table [id=098052b285984de7b615247d7c3908e6]) successfully deleted
I20260927 00:46:57.952725 2630 tablet_bootstrap.cc:492] T 6af7137078354672a09a98268e1953b5 P c84e3dfe390e42589ab1c1d7731889fc: No bootstrap required, opened a new log
I20260927 00:46:57.952826 2630 ts_tablet_manager.cc:1397] T 6af7137078354672a09a98268e1953b5 P c84e3dfe390e42589ab1c1d7731889fc: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260927 00:46:57.953256 2631 tablet_bootstrap.cc:492] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735: No bootstrap required, opened a new log
I20260927 00:46:57.953339 2631 ts_tablet_manager.cc:1397] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:46:57.953375 2630 raft_consensus.cc:348] T 6af7137078354672a09a98268e1953b5 P c84e3dfe390e42589ab1c1d7731889fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:57.953510 2630 raft_consensus.cc:374] T 6af7137078354672a09a98268e1953b5 P c84e3dfe390e42589ab1c1d7731889fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:57.953544 2630 raft_consensus.cc:729] T 6af7137078354672a09a98268e1953b5 P c84e3dfe390e42589ab1c1d7731889fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84e3dfe390e42589ab1c1d7731889fc, State: Initialized, Role: FOLLOWER
I20260927 00:46:57.953679 2630 consensus_queue.cc:260] T 6af7137078354672a09a98268e1953b5 P c84e3dfe390e42589ab1c1d7731889fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:57.953858 2631 raft_consensus.cc:348] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:57.953969 2630 ts_tablet_manager.cc:1428] T 6af7137078354672a09a98268e1953b5 P c84e3dfe390e42589ab1c1d7731889fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260927 00:46:57.953980 2631 raft_consensus.cc:374] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:57.954013 2631 raft_consensus.cc:729] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d30b0c6d65d4dd1b23f9742e7ea1735, State: Initialized, Role: FOLLOWER
I20260927 00:46:57.954140 2631 consensus_queue.cc:260] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:57.954424 2631 ts_tablet_manager.cc:1428] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:46:57.954741 2632 tablet_bootstrap.cc:492] T 6af7137078354672a09a98268e1953b5 P 3dd218cc6b694cd485ad6e25b5b73547: Bootstrap starting.
I20260927 00:46:57.955927 2632 tablet_bootstrap.cc:654] T 6af7137078354672a09a98268e1953b5 P 3dd218cc6b694cd485ad6e25b5b73547: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:57.957427 2632 tablet_bootstrap.cc:492] T 6af7137078354672a09a98268e1953b5 P 3dd218cc6b694cd485ad6e25b5b73547: No bootstrap required, opened a new log
I20260927 00:46:57.957516 2632 ts_tablet_manager.cc:1397] T 6af7137078354672a09a98268e1953b5 P 3dd218cc6b694cd485ad6e25b5b73547: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:46:57.958001 2632 raft_consensus.cc:348] T 6af7137078354672a09a98268e1953b5 P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:57.958146 2632 raft_consensus.cc:374] T 6af7137078354672a09a98268e1953b5 P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:57.958194 2632 raft_consensus.cc:729] T 6af7137078354672a09a98268e1953b5 P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd218cc6b694cd485ad6e25b5b73547, State: Initialized, Role: FOLLOWER
I20260927 00:46:57.958339 2632 consensus_queue.cc:260] T 6af7137078354672a09a98268e1953b5 P 3dd218cc6b694cd485ad6e25b5b73547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:57.958645 2632 ts_tablet_manager.cc:1428] T 6af7137078354672a09a98268e1953b5 P 3dd218cc6b694cd485ad6e25b5b73547: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:46:58.213838 2636 raft_consensus.cc:482] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:46:58.214079 2636 raft_consensus.cc:504] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:58.215633 2636 leader_election.cc:288] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c84e3dfe390e42589ab1c1d7731889fc (127.0.98.1:40547), 3dd218cc6b694cd485ad6e25b5b73547 (127.0.98.3:40181)
I20260927 00:46:58.220211 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.216219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49085 (local address 127.0.98.3:40181)
0927 00:46:58.216450 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:58.216457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:58.216587 (+ 130us) server_negotiation.cc:407] Connection header received
0927 00:46:58.216819 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:58.216822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:58.216869 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:58.216951 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:58.217288 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.217859 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.219103 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.219566 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.219704 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.219789 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:58.219962 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 00:46:58.220050 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 00:46:58.220263 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.215948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40181 (local address 127.0.98.2:49085)
0927 00:46:58.216447 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:58.216465 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:58.216671 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:58.216974 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:58.216977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:58.216994 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:58.217167 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:58.217175 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.217981 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.217987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:58.218968 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:58.218977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.219720 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.219723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:58.219813 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.219842 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 00:46:58.219943 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 00:46:58.220034 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":28,"spinlock_wait_cycles":13696,"thread_start_us":112,"threads_started":1}
I20260927 00:46:58.220562 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.215943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40547 (local address 127.0.98.2:50887)
0927 00:46:58.216538 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:58.216551 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:58.216705 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:58.217040 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:58.217043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:58.217062 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:46:58.217222 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:58.217228 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.218077 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.218082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:58.218986 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:58.218997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.220052 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.220056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:58.220176 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.220202 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:46:58.220321 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 00:46:58.220393 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":48,"thread_start_us":123,"threads_started":1}
I20260927 00:46:58.220762 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.216087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50887 (local address 127.0.98.1:40547)
0927 00:46:58.216457 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:58.216479 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:58.216649 (+ 170us) server_negotiation.cc:407] Connection header received
0927 00:46:58.216846 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:58.216850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:58.216914 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:58.217033 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:46:58.217353 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.217933 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.219382 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.219882 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.220018 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.220096 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:58.220338 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 00:46:58.220571 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":95,"threads_started":1}
I20260927 00:46:58.221287 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af7137078354672a09a98268e1953b5" candidate_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84e3dfe390e42589ab1c1d7731889fc" is_pre_election: true
I20260927 00:46:58.221370 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af7137078354672a09a98268e1953b5" candidate_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dd218cc6b694cd485ad6e25b5b73547" is_pre_election: true
I20260927 00:46:58.221500 2280 raft_consensus.cc:2454] T 6af7137078354672a09a98268e1953b5 P c84e3dfe390e42589ab1c1d7731889fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d30b0c6d65d4dd1b23f9742e7ea1735 in term 0.
I20260927 00:46:58.221750 2561 raft_consensus.cc:2454] T 6af7137078354672a09a98268e1953b5 P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d30b0c6d65d4dd1b23f9742e7ea1735 in term 0.
I20260927 00:46:58.222020 2355 leader_election.cc:302] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d30b0c6d65d4dd1b23f9742e7ea1735, c84e3dfe390e42589ab1c1d7731889fc; no voters:
I20260927 00:46:58.222409 2636 raft_consensus.cc:2781] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:46:58.222507 2636 raft_consensus.cc:482] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:46:58.222568 2636 raft_consensus.cc:3037] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:58.224030 2636 raft_consensus.cc:504] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:58.224583 2636 leader_election.cc:288] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [CANDIDATE]: Term 1 election: Requested vote from peers c84e3dfe390e42589ab1c1d7731889fc (127.0.98.1:40547), 3dd218cc6b694cd485ad6e25b5b73547 (127.0.98.3:40181)
I20260927 00:46:58.225028 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af7137078354672a09a98268e1953b5" candidate_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dd218cc6b694cd485ad6e25b5b73547"
I20260927 00:46:58.225065 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af7137078354672a09a98268e1953b5" candidate_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84e3dfe390e42589ab1c1d7731889fc"
I20260927 00:46:58.225195 2561 raft_consensus.cc:3037] T 6af7137078354672a09a98268e1953b5 P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:58.225225 2280 raft_consensus.cc:3037] T 6af7137078354672a09a98268e1953b5 P c84e3dfe390e42589ab1c1d7731889fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:46:58.226466 2280 raft_consensus.cc:2454] T 6af7137078354672a09a98268e1953b5 P c84e3dfe390e42589ab1c1d7731889fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d30b0c6d65d4dd1b23f9742e7ea1735 in term 1.
I20260927 00:46:58.226465 2561 raft_consensus.cc:2454] T 6af7137078354672a09a98268e1953b5 P 3dd218cc6b694cd485ad6e25b5b73547 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d30b0c6d65d4dd1b23f9742e7ea1735 in term 1.
I20260927 00:46:58.227330 2354 leader_election.cc:302] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dd218cc6b694cd485ad6e25b5b73547, 4d30b0c6d65d4dd1b23f9742e7ea1735; no voters:
I20260927 00:46:58.227555 2636 raft_consensus.cc:2781] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:46:58.227784 2636 raft_consensus.cc:686] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 1 LEADER]: Becoming Leader. State: Replica: 4d30b0c6d65d4dd1b23f9742e7ea1735, State: Running, Role: LEADER
I20260927 00:46:58.228166 2636 consensus_queue.cc:237] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:58.231220 2124 catalog_manager.cc:5495] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d30b0c6d65d4dd1b23f9742e7ea1735 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } health_report { overall_health: UNKNOWN } } }
I20260927 00:46:58.275576 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.269573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40262 (local address 127.0.98.2:43073)
0927 00:46:58.270054 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:58.270445 (+ 391us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:58.270473 (+ 28us) server_negotiation.cc:407] Connection header received
0927 00:46:58.270586 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:58.270590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:58.270753 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:58.270961 (+ 208us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:46:58.271450 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.272297 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.273526 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.273812 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.273934 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.274339 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:46:58.275045 (+ 706us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:46:58.275163 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:58.275299 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 00:46:58.275383 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":185,"threads_started":1}
I20260927 00:46:58.290390 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.286480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40985 (local address 127.0.98.3:40181)
0927 00:46:58.286658 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:58.286664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:58.286862 (+ 198us) server_negotiation.cc:407] Connection header received
0927 00:46:58.287073 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:58.287076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:58.287140 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:58.287234 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:46:58.287545 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.288156 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.289155 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.289391 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.289506 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.289862 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:46:58.289955 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:46:58.290038 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:58.290131 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 00:46:58.290192 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 00:46:58.290398 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.286310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40181 (local address 127.0.98.2:40985)
0927 00:46:58.286765 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:58.286779 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:58.286936 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:58.287249 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:58.287253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:58.287262 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:46:58.287428 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:58.287434 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.288334 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.288338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:58.289027 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:58.289035 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.289518 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.289521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:58.289637 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.289703 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:46:58.290042 (+ 339us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:46:58.290052 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 00:46:58.290120 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 00:46:58.290216 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":141,"threads_started":1}
I20260927 00:46:58.297981 2280 raft_consensus.cc:1261] T 6af7137078354672a09a98268e1953b5 P c84e3dfe390e42589ab1c1d7731889fc [term 1 FOLLOWER]: Refusing update from remote peer 4d30b0c6d65d4dd1b23f9742e7ea1735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:46:58.298272 2561 raft_consensus.cc:1261] T 6af7137078354672a09a98268e1953b5 P 3dd218cc6b694cd485ad6e25b5b73547 [term 1 FOLLOWER]: Refusing update from remote peer 4d30b0c6d65d4dd1b23f9742e7ea1735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:46:58.298591 2636 consensus_queue.cc:1035] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:46:58.298861 2680 consensus_queue.cc:1035] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:46:58.319839 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.316299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60699 (local address 127.0.98.2:43073)
0927 00:46:58.316408 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:58.316415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:58.316452 (+ 37us) server_negotiation.cc:407] Connection header received
0927 00:46:58.316624 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:58.316627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:58.316690 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:58.316796 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:46:58.317110 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.317815 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.318711 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.318943 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.319035 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.319315 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:46:58.319397 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:46:58.319455 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:58.319599 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 00:46:58.319672 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 00:46:58.319839 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:58.316158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43073 (local address 127.0.98.3:60699)
0927 00:46:58.316383 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 00:46:58.316401 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:46:58.316505 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:46:58.316822 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:46:58.316824 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:46:58.316833 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:46:58.316975 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:58.316982 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.317925 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.317931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:58.318577 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:46:58.318585 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:46:58.319063 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:46:58.319066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:46:58.319156 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:58.319195 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:46:58.319467 (+ 272us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:46:58.319477 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 00:46:58.319552 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 00:46:58.319653 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
W20260927 00:46:58.320793 2657 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 00:46:59.911979 2124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:46:59.912220 2124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:46:59.915763 2124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ef1885acaaa4265ab30b56cd692d00e: Sending DeleteTablet for 3 replicas of tablet 6af7137078354672a09a98268e1953b5
I20260927 00:46:59.916760 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet 6af7137078354672a09a98268e1953b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:46:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40260
I20260927 00:46:59.916941 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 6af7137078354672a09a98268e1953b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:46:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36662
I20260927 00:46:59.917259 2712 tablet_replica.cc:330] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735: stopping tablet replica
I20260927 00:46:59.917466 2712 raft_consensus.cc:2229] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:46:59.918174 2712 raft_consensus.cc:2258] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:46:59.918838 2713 tablet_replica.cc:330] T 6af7137078354672a09a98268e1953b5 P 3dd218cc6b694cd485ad6e25b5b73547: stopping tablet replica
I20260927 00:46:59.919023 2713 raft_consensus.cc:2229] T 6af7137078354672a09a98268e1953b5 P 3dd218cc6b694cd485ad6e25b5b73547 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:46:59.919195 2713 raft_consensus.cc:2258] T 6af7137078354672a09a98268e1953b5 P 3dd218cc6b694cd485ad6e25b5b73547 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:46:59.919183 2712 ts_tablet_manager.cc:1905] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:46:59.921016 2261 tablet_service.cc:1514] Processing DeleteTablet for tablet 6af7137078354672a09a98268e1953b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:46:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58764
I20260927 00:46:59.921463 2715 tablet_replica.cc:330] T 6af7137078354672a09a98268e1953b5 P c84e3dfe390e42589ab1c1d7731889fc: stopping tablet replica
I20260927 00:46:59.921679 2715 raft_consensus.cc:2229] T 6af7137078354672a09a98268e1953b5 P c84e3dfe390e42589ab1c1d7731889fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:46:59.921839 2715 raft_consensus.cc:2258] T 6af7137078354672a09a98268e1953b5 P c84e3dfe390e42589ab1c1d7731889fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:46:59.922282 2713 ts_tablet_manager.cc:1905] T 6af7137078354672a09a98268e1953b5 P 3dd218cc6b694cd485ad6e25b5b73547: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:46:59.922528 2715 ts_tablet_manager.cc:1905] T 6af7137078354672a09a98268e1953b5 P c84e3dfe390e42589ab1c1d7731889fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:46:59.925736 2715 ts_tablet_manager.cc:1918] T 6af7137078354672a09a98268e1953b5 P c84e3dfe390e42589ab1c1d7731889fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 00:46:59.925812 2713 ts_tablet_manager.cc:1918] T 6af7137078354672a09a98268e1953b5 P 3dd218cc6b694cd485ad6e25b5b73547: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 00:46:59.925848 2715 log.cc:1196] T 6af7137078354672a09a98268e1953b5 P c84e3dfe390e42589ab1c1d7731889fc: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-0/wal/wals/6af7137078354672a09a98268e1953b5
I20260927 00:46:59.925889 2713 log.cc:1196] T 6af7137078354672a09a98268e1953b5 P 3dd218cc6b694cd485ad6e25b5b73547: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-2/wal/wals/6af7137078354672a09a98268e1953b5
I20260927 00:46:59.925858 2712 ts_tablet_manager.cc:1918] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 00:46:59.925959 2712 log.cc:1196] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-1/wal/wals/6af7137078354672a09a98268e1953b5
I20260927 00:46:59.926167 2715 ts_tablet_manager.cc:1939] T 6af7137078354672a09a98268e1953b5 P c84e3dfe390e42589ab1c1d7731889fc: Deleting consensus metadata
I20260927 00:46:59.926204 2713 ts_tablet_manager.cc:1939] T 6af7137078354672a09a98268e1953b5 P 3dd218cc6b694cd485ad6e25b5b73547: Deleting consensus metadata
I20260927 00:46:59.926393 2712 ts_tablet_manager.cc:1939] T 6af7137078354672a09a98268e1953b5 P 4d30b0c6d65d4dd1b23f9742e7ea1735: Deleting consensus metadata
I20260927 00:46:59.927327 2111 catalog_manager.cc:4841] TS c84e3dfe390e42589ab1c1d7731889fc (127.0.98.1:40547): tablet 6af7137078354672a09a98268e1953b5 (table test-table [id=97c4dd1feb924fa29e4cf14197b3c392]) successfully deleted
I20260927 00:46:59.927656 2110 catalog_manager.cc:4841] TS 4d30b0c6d65d4dd1b23f9742e7ea1735 (127.0.98.2:43073): tablet 6af7137078354672a09a98268e1953b5 (table test-table [id=97c4dd1feb924fa29e4cf14197b3c392]) successfully deleted
I20260927 00:46:59.927840 2110 catalog_manager.cc:4841] TS 3dd218cc6b694cd485ad6e25b5b73547 (127.0.98.3:40181): tablet 6af7137078354672a09a98268e1953b5 (table test-table [id=97c4dd1feb924fa29e4cf14197b3c392]) successfully deleted
Sep 27 00:46:59 dist-test-slave-86rs krb5kdc[2079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470019, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:46:59 dist-test-slave-86rs krb5kdc[2079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470019, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:46:59.962741 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:46:59.944372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54240 (local address 127.0.98.62:43249)
0927 00:46:59.944699 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0927 00:46:59.944711 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:46:59.944812 (+ 101us) server_negotiation.cc:407] Connection header received
0927 00:46:59.944970 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:46:59.944972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:46:59.945017 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:46:59.945119 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:46:59.946088 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:59.946760 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:59.947729 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:46:59.948031 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:46:59.948137 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:46:59.957530 (+ 9393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:46:59.957535 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:46:59.957537 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:46:59.957586 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:46:59.959821 (+ 2235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:59.960681 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:59.960685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:59.960687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:59.960827 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:46:59.961447 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:46:59.961451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:46:59.961453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:46:59.961898 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:46:59.962053 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 00:46:59.962404 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 00:46:59.962559 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":146,"threads_started":1}
I20260927 00:46:59.966367 2124 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:46:59.967007 2124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:46:59.973157 2540 tablet_service.cc:1467] Processing CreateTablet for tablet d59daf3de52941e588aae01ee6903c64 (DEFAULT_TABLE table=test-table [id=9014813bc47e4888a9de983a289737ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:59.973161 2401 tablet_service.cc:1467] Processing CreateTablet for tablet d59daf3de52941e588aae01ee6903c64 (DEFAULT_TABLE table=test-table [id=9014813bc47e4888a9de983a289737ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:59.973122 2261 tablet_service.cc:1467] Processing CreateTablet for tablet d59daf3de52941e588aae01ee6903c64 (DEFAULT_TABLE table=test-table [id=9014813bc47e4888a9de983a289737ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:46:59.973511 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d59daf3de52941e588aae01ee6903c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:59.973511 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d59daf3de52941e588aae01ee6903c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:59.973511 2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d59daf3de52941e588aae01ee6903c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:46:59.976037 2723 tablet_bootstrap.cc:492] T d59daf3de52941e588aae01ee6903c64 P 4d30b0c6d65d4dd1b23f9742e7ea1735: Bootstrap starting.
I20260927 00:46:59.977116 2725 tablet_bootstrap.cc:492] T d59daf3de52941e588aae01ee6903c64 P c84e3dfe390e42589ab1c1d7731889fc: Bootstrap starting.
I20260927 00:46:59.977121 2724 tablet_bootstrap.cc:492] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547: Bootstrap starting.
I20260927 00:46:59.977599 2723 tablet_bootstrap.cc:654] T d59daf3de52941e588aae01ee6903c64 P 4d30b0c6d65d4dd1b23f9742e7ea1735: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:59.978299 2724 tablet_bootstrap.cc:654] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:59.978299 2725 tablet_bootstrap.cc:654] T d59daf3de52941e588aae01ee6903c64 P c84e3dfe390e42589ab1c1d7731889fc: Neither blocks nor log segments found. Creating new log.
I20260927 00:46:59.979081 2723 tablet_bootstrap.cc:492] T d59daf3de52941e588aae01ee6903c64 P 4d30b0c6d65d4dd1b23f9742e7ea1735: No bootstrap required, opened a new log
I20260927 00:46:59.979206 2723 ts_tablet_manager.cc:1397] T d59daf3de52941e588aae01ee6903c64 P 4d30b0c6d65d4dd1b23f9742e7ea1735: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:46:59.979779 2724 tablet_bootstrap.cc:492] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547: No bootstrap required, opened a new log
I20260927 00:46:59.979893 2724 ts_tablet_manager.cc:1397] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:46:59.979864 2723 raft_consensus.cc:348] T d59daf3de52941e588aae01ee6903c64 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:59.980023 2723 raft_consensus.cc:374] T d59daf3de52941e588aae01ee6903c64 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:59.980072 2723 raft_consensus.cc:729] T d59daf3de52941e588aae01ee6903c64 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d30b0c6d65d4dd1b23f9742e7ea1735, State: Initialized, Role: FOLLOWER
I20260927 00:46:59.980254 2723 consensus_queue.cc:260] T d59daf3de52941e588aae01ee6903c64 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:59.980436 2724 raft_consensus.cc:348] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:59.980587 2724 raft_consensus.cc:374] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:59.980631 2724 raft_consensus.cc:729] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd218cc6b694cd485ad6e25b5b73547, State: Initialized, Role: FOLLOWER
I20260927 00:46:59.980822 2724 consensus_queue.cc:260] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:59.980995 2725 tablet_bootstrap.cc:492] T d59daf3de52941e588aae01ee6903c64 P c84e3dfe390e42589ab1c1d7731889fc: No bootstrap required, opened a new log
I20260927 00:46:59.981089 2725 ts_tablet_manager.cc:1397] T d59daf3de52941e588aae01ee6903c64 P c84e3dfe390e42589ab1c1d7731889fc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 00:46:59.981163 2724 ts_tablet_manager.cc:1428] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:46:59.981629 2725 raft_consensus.cc:348] T d59daf3de52941e588aae01ee6903c64 P c84e3dfe390e42589ab1c1d7731889fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:59.981772 2725 raft_consensus.cc:374] T d59daf3de52941e588aae01ee6903c64 P c84e3dfe390e42589ab1c1d7731889fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:46:59.981815 2725 raft_consensus.cc:729] T d59daf3de52941e588aae01ee6903c64 P c84e3dfe390e42589ab1c1d7731889fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84e3dfe390e42589ab1c1d7731889fc, State: Initialized, Role: FOLLOWER
I20260927 00:46:59.981961 2725 consensus_queue.cc:260] T d59daf3de52941e588aae01ee6903c64 P c84e3dfe390e42589ab1c1d7731889fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:46:59.982427 2725 ts_tablet_manager.cc:1428] T d59daf3de52941e588aae01ee6903c64 P c84e3dfe390e42589ab1c1d7731889fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:46:59.983107 2723 ts_tablet_manager.cc:1428] T d59daf3de52941e588aae01ee6903c64 P 4d30b0c6d65d4dd1b23f9742e7ea1735: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 00:47:00.050727 2640 raft_consensus.cc:482] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:47:00.050944 2640 raft_consensus.cc:504] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:47:00.051438 2640 leader_election.cc:288] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c84e3dfe390e42589ab1c1d7731889fc (127.0.98.1:40547), 4d30b0c6d65d4dd1b23f9742e7ea1735 (127.0.98.2:43073)
I20260927 00:47:00.052074 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d59daf3de52941e588aae01ee6903c64" candidate_uuid: "3dd218cc6b694cd485ad6e25b5b73547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84e3dfe390e42589ab1c1d7731889fc" is_pre_election: true
I20260927 00:47:00.052064 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d59daf3de52941e588aae01ee6903c64" candidate_uuid: "3dd218cc6b694cd485ad6e25b5b73547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" is_pre_election: true
I20260927 00:47:00.052311 2280 raft_consensus.cc:2454] T d59daf3de52941e588aae01ee6903c64 P c84e3dfe390e42589ab1c1d7731889fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dd218cc6b694cd485ad6e25b5b73547 in term 0.
I20260927 00:47:00.052323 2421 raft_consensus.cc:2454] T d59daf3de52941e588aae01ee6903c64 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dd218cc6b694cd485ad6e25b5b73547 in term 0.
I20260927 00:47:00.052748 2495 leader_election.cc:302] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dd218cc6b694cd485ad6e25b5b73547, c84e3dfe390e42589ab1c1d7731889fc; no voters:
I20260927 00:47:00.052969 2640 raft_consensus.cc:2781] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:47:00.053046 2640 raft_consensus.cc:482] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:47:00.053099 2640 raft_consensus.cc:3037] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:00.054544 2640 raft_consensus.cc:504] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:47:00.055049 2640 leader_election.cc:288] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547 [CANDIDATE]: Term 1 election: Requested vote from peers c84e3dfe390e42589ab1c1d7731889fc (127.0.98.1:40547), 4d30b0c6d65d4dd1b23f9742e7ea1735 (127.0.98.2:43073)
I20260927 00:47:00.055497 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d59daf3de52941e588aae01ee6903c64" candidate_uuid: "3dd218cc6b694cd485ad6e25b5b73547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735"
I20260927 00:47:00.055500 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d59daf3de52941e588aae01ee6903c64" candidate_uuid: "3dd218cc6b694cd485ad6e25b5b73547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84e3dfe390e42589ab1c1d7731889fc"
I20260927 00:47:00.055658 2421 raft_consensus.cc:3037] T d59daf3de52941e588aae01ee6903c64 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:00.055673 2280 raft_consensus.cc:3037] T d59daf3de52941e588aae01ee6903c64 P c84e3dfe390e42589ab1c1d7731889fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:00.056905 2280 raft_consensus.cc:2454] T d59daf3de52941e588aae01ee6903c64 P c84e3dfe390e42589ab1c1d7731889fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dd218cc6b694cd485ad6e25b5b73547 in term 1.
I20260927 00:47:00.056906 2421 raft_consensus.cc:2454] T d59daf3de52941e588aae01ee6903c64 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dd218cc6b694cd485ad6e25b5b73547 in term 1.
I20260927 00:47:00.057272 2495 leader_election.cc:302] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dd218cc6b694cd485ad6e25b5b73547, c84e3dfe390e42589ab1c1d7731889fc; no voters:
I20260927 00:47:00.057461 2640 raft_consensus.cc:2781] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:00.057557 2640 raft_consensus.cc:686] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547 [term 1 LEADER]: Becoming Leader. State: Replica: 3dd218cc6b694cd485ad6e25b5b73547, State: Running, Role: LEADER
I20260927 00:47:00.057745 2640 consensus_queue.cc:237] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:47:00.059911 2122 catalog_manager.cc:5495] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3dd218cc6b694cd485ad6e25b5b73547 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3dd218cc6b694cd485ad6e25b5b73547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } health_report { overall_health: HEALTHY } } }
I20260927 00:47:00.099408 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:00.095280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36690 (local address 127.0.98.3:40181)
0927 00:47:00.095621 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:00.095647 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:00.095659 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:47:00.095705 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:00.095707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:00.095758 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:00.095874 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:47:00.096175 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:00.097129 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:00.098138 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:00.098386 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:00.098507 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:00.098815 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:47:00.098917 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:47:00.098998 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:00.099132 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 00:47:00.099202 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":124,"threads_started":1}
I20260927 00:47:00.109054 2280 raft_consensus.cc:1261] T d59daf3de52941e588aae01ee6903c64 P c84e3dfe390e42589ab1c1d7731889fc [term 1 FOLLOWER]: Refusing update from remote peer 3dd218cc6b694cd485ad6e25b5b73547: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:47:00.109052 2421 raft_consensus.cc:1261] T d59daf3de52941e588aae01ee6903c64 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 1 FOLLOWER]: Refusing update from remote peer 3dd218cc6b694cd485ad6e25b5b73547: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:47:00.109511 2693 consensus_queue.cc:1035] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:47:00.109728 2640 consensus_queue.cc:1035] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:47:00.148747 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:47:00.148969 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:47:00.152171 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ef1885acaaa4265ab30b56cd692d00e: Sending DeleteTablet for 3 replicas of tablet d59daf3de52941e588aae01ee6903c64
I20260927 00:47:00.153365 2261 tablet_service.cc:1514] Processing DeleteTablet for tablet d59daf3de52941e588aae01ee6903c64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58764
I20260927 00:47:00.153376 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet d59daf3de52941e588aae01ee6903c64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36662
I20260927 00:47:00.153460 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet d59daf3de52941e588aae01ee6903c64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40260
I20260927 00:47:00.153679 2715 tablet_replica.cc:330] T d59daf3de52941e588aae01ee6903c64 P c84e3dfe390e42589ab1c1d7731889fc: stopping tablet replica
I20260927 00:47:00.153684 2713 tablet_replica.cc:330] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547: stopping tablet replica
I20260927 00:47:00.153710 2712 tablet_replica.cc:330] T d59daf3de52941e588aae01ee6903c64 P 4d30b0c6d65d4dd1b23f9742e7ea1735: stopping tablet replica
I20260927 00:47:00.153838 2715 raft_consensus.cc:2229] T d59daf3de52941e588aae01ee6903c64 P c84e3dfe390e42589ab1c1d7731889fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:47:00.153839 2713 raft_consensus.cc:2229] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:47:00.153844 2712 raft_consensus.cc:2229] T d59daf3de52941e588aae01ee6903c64 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:47:00.154003 2712 raft_consensus.cc:2258] T d59daf3de52941e588aae01ee6903c64 P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:00.154006 2715 raft_consensus.cc:2258] T d59daf3de52941e588aae01ee6903c64 P c84e3dfe390e42589ab1c1d7731889fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:00.154146 2713 raft_consensus.cc:2258] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:00.155215 2713 ts_tablet_manager.cc:1905] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:00.155216 2715 ts_tablet_manager.cc:1905] T d59daf3de52941e588aae01ee6903c64 P c84e3dfe390e42589ab1c1d7731889fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:00.155135 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 00:47:00.155503 2712 ts_tablet_manager.cc:1905] T d59daf3de52941e588aae01ee6903c64 P 4d30b0c6d65d4dd1b23f9742e7ea1735: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 00:47:00.155820 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:47:00.158457 2713 ts_tablet_manager.cc:1918] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:47:00.158536 2713 log.cc:1196] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-2/wal/wals/d59daf3de52941e588aae01ee6903c64
I20260927 00:47:00.158783 2713 ts_tablet_manager.cc:1939] T d59daf3de52941e588aae01ee6903c64 P 3dd218cc6b694cd485ad6e25b5b73547: Deleting consensus metadata
I20260927 00:47:00.158740 2715 ts_tablet_manager.cc:1918] T d59daf3de52941e588aae01ee6903c64 P c84e3dfe390e42589ab1c1d7731889fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:47:00.158845 2715 log.cc:1196] T d59daf3de52941e588aae01ee6903c64 P c84e3dfe390e42589ab1c1d7731889fc: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-0/wal/wals/d59daf3de52941e588aae01ee6903c64
I20260927 00:47:00.158947 2712 ts_tablet_manager.cc:1918] T d59daf3de52941e588aae01ee6903c64 P 4d30b0c6d65d4dd1b23f9742e7ea1735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:47:00.159039 2712 log.cc:1196] T d59daf3de52941e588aae01ee6903c64 P 4d30b0c6d65d4dd1b23f9742e7ea1735: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-1/wal/wals/d59daf3de52941e588aae01ee6903c64
I20260927 00:47:00.159149 2715 ts_tablet_manager.cc:1939] T d59daf3de52941e588aae01ee6903c64 P c84e3dfe390e42589ab1c1d7731889fc: Deleting consensus metadata
I20260927 00:47:00.159340 2712 ts_tablet_manager.cc:1939] T d59daf3de52941e588aae01ee6903c64 P 4d30b0c6d65d4dd1b23f9742e7ea1735: Deleting consensus metadata
I20260927 00:47:00.160130 2111 catalog_manager.cc:4841] TS c84e3dfe390e42589ab1c1d7731889fc (127.0.98.1:40547): tablet d59daf3de52941e588aae01ee6903c64 (table test-table [id=9014813bc47e4888a9de983a289737ed]) successfully deleted
I20260927 00:47:00.160164 2110 catalog_manager.cc:4841] TS 4d30b0c6d65d4dd1b23f9742e7ea1735 (127.0.98.2:43073): tablet d59daf3de52941e588aae01ee6903c64 (table test-table [id=9014813bc47e4888a9de983a289737ed]) successfully deleted
I20260927 00:47:00.160538 2110 catalog_manager.cc:4841] TS 3dd218cc6b694cd485ad6e25b5b73547 (127.0.98.3:40181): tablet d59daf3de52941e588aae01ee6903c64 (table test-table [id=9014813bc47e4888a9de983a289737ed]) successfully deleted
I20260927 00:47:00.165534 2401 tablet_service.cc:1467] Processing CreateTablet for tablet 950ba63893504c97b20e5b7fcd6df39f (DEFAULT_TABLE table=test-table [id=302bb82ed9a44a9a9705e0755c455b2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:00.165534 2261 tablet_service.cc:1467] Processing CreateTablet for tablet 950ba63893504c97b20e5b7fcd6df39f (DEFAULT_TABLE table=test-table [id=302bb82ed9a44a9a9705e0755c455b2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:00.165588 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 950ba63893504c97b20e5b7fcd6df39f (DEFAULT_TABLE table=test-table [id=302bb82ed9a44a9a9705e0755c455b2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:00.165963 2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 950ba63893504c97b20e5b7fcd6df39f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:00.165962 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 950ba63893504c97b20e5b7fcd6df39f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:00.165966 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 950ba63893504c97b20e5b7fcd6df39f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:00.168560 2724 tablet_bootstrap.cc:492] T 950ba63893504c97b20e5b7fcd6df39f P 3dd218cc6b694cd485ad6e25b5b73547: Bootstrap starting.
I20260927 00:47:00.168787 2725 tablet_bootstrap.cc:492] T 950ba63893504c97b20e5b7fcd6df39f P c84e3dfe390e42589ab1c1d7731889fc: Bootstrap starting.
I20260927 00:47:00.169463 2723 tablet_bootstrap.cc:492] T 950ba63893504c97b20e5b7fcd6df39f P 4d30b0c6d65d4dd1b23f9742e7ea1735: Bootstrap starting.
I20260927 00:47:00.169688 2724 tablet_bootstrap.cc:654] T 950ba63893504c97b20e5b7fcd6df39f P 3dd218cc6b694cd485ad6e25b5b73547: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:00.169911 2725 tablet_bootstrap.cc:654] T 950ba63893504c97b20e5b7fcd6df39f P c84e3dfe390e42589ab1c1d7731889fc: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:00.170981 2723 tablet_bootstrap.cc:654] T 950ba63893504c97b20e5b7fcd6df39f P 4d30b0c6d65d4dd1b23f9742e7ea1735: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:00.171047 2724 tablet_bootstrap.cc:492] T 950ba63893504c97b20e5b7fcd6df39f P 3dd218cc6b694cd485ad6e25b5b73547: No bootstrap required, opened a new log
I20260927 00:47:00.171139 2724 ts_tablet_manager.cc:1397] T 950ba63893504c97b20e5b7fcd6df39f P 3dd218cc6b694cd485ad6e25b5b73547: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:47:00.171671 2725 tablet_bootstrap.cc:492] T 950ba63893504c97b20e5b7fcd6df39f P c84e3dfe390e42589ab1c1d7731889fc: No bootstrap required, opened a new log
I20260927 00:47:00.171772 2725 ts_tablet_manager.cc:1397] T 950ba63893504c97b20e5b7fcd6df39f P c84e3dfe390e42589ab1c1d7731889fc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:47:00.171741 2724 raft_consensus.cc:348] T 950ba63893504c97b20e5b7fcd6df39f P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:47:00.171880 2724 raft_consensus.cc:374] T 950ba63893504c97b20e5b7fcd6df39f P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:00.171917 2724 raft_consensus.cc:729] T 950ba63893504c97b20e5b7fcd6df39f P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd218cc6b694cd485ad6e25b5b73547, State: Initialized, Role: FOLLOWER
I20260927 00:47:00.172371 2724 consensus_queue.cc:260] T 950ba63893504c97b20e5b7fcd6df39f P 3dd218cc6b694cd485ad6e25b5b73547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:47:00.172394 2725 raft_consensus.cc:348] T 950ba63893504c97b20e5b7fcd6df39f P c84e3dfe390e42589ab1c1d7731889fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:47:00.172546 2725 raft_consensus.cc:374] T 950ba63893504c97b20e5b7fcd6df39f P c84e3dfe390e42589ab1c1d7731889fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:00.172585 2725 raft_consensus.cc:729] T 950ba63893504c97b20e5b7fcd6df39f P c84e3dfe390e42589ab1c1d7731889fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84e3dfe390e42589ab1c1d7731889fc, State: Initialized, Role: FOLLOWER
I20260927 00:47:00.172729 2724 ts_tablet_manager.cc:1428] T 950ba63893504c97b20e5b7fcd6df39f P 3dd218cc6b694cd485ad6e25b5b73547: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:47:00.172741 2725 consensus_queue.cc:260] T 950ba63893504c97b20e5b7fcd6df39f P c84e3dfe390e42589ab1c1d7731889fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:47:00.173098 2725 ts_tablet_manager.cc:1428] T 950ba63893504c97b20e5b7fcd6df39f P c84e3dfe390e42589ab1c1d7731889fc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 00:47:00.173333 2723 tablet_bootstrap.cc:492] T 950ba63893504c97b20e5b7fcd6df39f P 4d30b0c6d65d4dd1b23f9742e7ea1735: No bootstrap required, opened a new log
I20260927 00:47:00.173439 2723 ts_tablet_manager.cc:1397] T 950ba63893504c97b20e5b7fcd6df39f P 4d30b0c6d65d4dd1b23f9742e7ea1735: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 00:47:00.174016 2723 raft_consensus.cc:348] T 950ba63893504c97b20e5b7fcd6df39f P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:47:00.174222 2723 raft_consensus.cc:374] T 950ba63893504c97b20e5b7fcd6df39f P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:00.174276 2723 raft_consensus.cc:729] T 950ba63893504c97b20e5b7fcd6df39f P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d30b0c6d65d4dd1b23f9742e7ea1735, State: Initialized, Role: FOLLOWER
I20260927 00:47:00.174461 2723 consensus_queue.cc:260] T 950ba63893504c97b20e5b7fcd6df39f P 4d30b0c6d65d4dd1b23f9742e7ea1735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:47:00.174835 2723 ts_tablet_manager.cc:1428] T 950ba63893504c97b20e5b7fcd6df39f P 4d30b0c6d65d4dd1b23f9742e7ea1735: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 00:47:00.327761 2690 raft_consensus.cc:482] T 950ba63893504c97b20e5b7fcd6df39f P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:47:00.327973 2690 raft_consensus.cc:504] T 950ba63893504c97b20e5b7fcd6df39f P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:47:00.328583 2690 leader_election.cc:288] T 950ba63893504c97b20e5b7fcd6df39f P 4d30b0c6d65d4dd1b23f9742e7ea1735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c84e3dfe390e42589ab1c1d7731889fc (127.0.98.1:40547), 3dd218cc6b694cd485ad6e25b5b73547 (127.0.98.3:40181)
I20260927 00:47:00.329157 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "950ba63893504c97b20e5b7fcd6df39f" candidate_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dd218cc6b694cd485ad6e25b5b73547" is_pre_election: true
I20260927 00:47:00.329169 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "950ba63893504c97b20e5b7fcd6df39f" candidate_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84e3dfe390e42589ab1c1d7731889fc" is_pre_election: true
I20260927 00:47:00.329357 2280 raft_consensus.cc:2454] T 950ba63893504c97b20e5b7fcd6df39f P c84e3dfe390e42589ab1c1d7731889fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d30b0c6d65d4dd1b23f9742e7ea1735 in term 0.
I20260927 00:47:00.329357 2561 raft_consensus.cc:2454] T 950ba63893504c97b20e5b7fcd6df39f P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d30b0c6d65d4dd1b23f9742e7ea1735 in term 0.
I20260927 00:47:00.329748 2354 leader_election.cc:302] T 950ba63893504c97b20e5b7fcd6df39f P 4d30b0c6d65d4dd1b23f9742e7ea1735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dd218cc6b694cd485ad6e25b5b73547, 4d30b0c6d65d4dd1b23f9742e7ea1735; no voters:
I20260927 00:47:00.329968 2690 raft_consensus.cc:2781] T 950ba63893504c97b20e5b7fcd6df39f P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:47:00.330050 2690 raft_consensus.cc:482] T 950ba63893504c97b20e5b7fcd6df39f P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:47:00.330125 2690 raft_consensus.cc:3037] T 950ba63893504c97b20e5b7fcd6df39f P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:00.331467 2690 raft_consensus.cc:504] T 950ba63893504c97b20e5b7fcd6df39f P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:47:00.331905 2690 leader_election.cc:288] T 950ba63893504c97b20e5b7fcd6df39f P 4d30b0c6d65d4dd1b23f9742e7ea1735 [CANDIDATE]: Term 1 election: Requested vote from peers c84e3dfe390e42589ab1c1d7731889fc (127.0.98.1:40547), 3dd218cc6b694cd485ad6e25b5b73547 (127.0.98.3:40181)
I20260927 00:47:00.332492 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "950ba63893504c97b20e5b7fcd6df39f" candidate_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84e3dfe390e42589ab1c1d7731889fc"
I20260927 00:47:00.332710 2280 raft_consensus.cc:3037] T 950ba63893504c97b20e5b7fcd6df39f P c84e3dfe390e42589ab1c1d7731889fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:00.333614 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "950ba63893504c97b20e5b7fcd6df39f" candidate_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dd218cc6b694cd485ad6e25b5b73547"
I20260927 00:47:00.333822 2561 raft_consensus.cc:3037] T 950ba63893504c97b20e5b7fcd6df39f P 3dd218cc6b694cd485ad6e25b5b73547 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:00.334482 2280 raft_consensus.cc:2454] T 950ba63893504c97b20e5b7fcd6df39f P c84e3dfe390e42589ab1c1d7731889fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d30b0c6d65d4dd1b23f9742e7ea1735 in term 1.
I20260927 00:47:00.334946 2355 leader_election.cc:302] T 950ba63893504c97b20e5b7fcd6df39f P 4d30b0c6d65d4dd1b23f9742e7ea1735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d30b0c6d65d4dd1b23f9742e7ea1735, c84e3dfe390e42589ab1c1d7731889fc; no voters:
I20260927 00:47:00.335224 2690 raft_consensus.cc:2781] T 950ba63893504c97b20e5b7fcd6df39f P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:00.335366 2690 raft_consensus.cc:686] T 950ba63893504c97b20e5b7fcd6df39f P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 1 LEADER]: Becoming Leader. State: Replica: 4d30b0c6d65d4dd1b23f9742e7ea1735, State: Running, Role: LEADER
I20260927 00:47:00.335433 2561 raft_consensus.cc:2454] T 950ba63893504c97b20e5b7fcd6df39f P 3dd218cc6b694cd485ad6e25b5b73547 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d30b0c6d65d4dd1b23f9742e7ea1735 in term 1.
I20260927 00:47:00.335626 2690 consensus_queue.cc:237] T 950ba63893504c97b20e5b7fcd6df39f P 4d30b0c6d65d4dd1b23f9742e7ea1735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } }
I20260927 00:47:00.338106 2122 catalog_manager.cc:5495] T 950ba63893504c97b20e5b7fcd6df39f P 4d30b0c6d65d4dd1b23f9742e7ea1735 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d30b0c6d65d4dd1b23f9742e7ea1735 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d30b0c6d65d4dd1b23f9742e7ea1735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 } health_report { overall_health: UNKNOWN } } }
I20260927 00:47:00.382226 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:00.377379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40264 (local address 127.0.98.2:43073)
0927 00:47:00.377752 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:00.377802 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:00.377822 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:47:00.377887 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:00.377894 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:00.377979 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:00.378088 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:47:00.378415 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:00.379414 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:00.380459 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:00.380961 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:00.381107 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:00.381399 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:47:00.381544 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:47:00.381636 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:00.381802 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 00:47:00.381899 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":140,"threads_started":1}
I20260927 00:47:00.390854 2280 raft_consensus.cc:1261] T 950ba63893504c97b20e5b7fcd6df39f P c84e3dfe390e42589ab1c1d7731889fc [term 1 FOLLOWER]: Refusing update from remote peer 4d30b0c6d65d4dd1b23f9742e7ea1735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:47:00.390914 2561 raft_consensus.cc:1261] T 950ba63893504c97b20e5b7fcd6df39f P 3dd218cc6b694cd485ad6e25b5b73547 [term 1 FOLLOWER]: Refusing update from remote peer 4d30b0c6d65d4dd1b23f9742e7ea1735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:47:00.391392 2706 consensus_queue.cc:1035] T 950ba63893504c97b20e5b7fcd6df39f P 4d30b0c6d65d4dd1b23f9742e7ea1735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dd218cc6b694cd485ad6e25b5b73547" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:47:00.391633 2690 consensus_queue.cc:1035] T 950ba63893504c97b20e5b7fcd6df39f P 4d30b0c6d65d4dd1b23f9742e7ea1735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c84e3dfe390e42589ab1c1d7731889fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 00:47:00.410586 2694 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 00:47:01.973881 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:47:01.974134 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:47:01.977341 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ef1885acaaa4265ab30b56cd692d00e: Sending DeleteTablet for 3 replicas of tablet 950ba63893504c97b20e5b7fcd6df39f
I20260927 00:47:01.978336 2261 tablet_service.cc:1514] Processing DeleteTablet for tablet 950ba63893504c97b20e5b7fcd6df39f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58764
I20260927 00:47:01.978546 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet 950ba63893504c97b20e5b7fcd6df39f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40260
I20260927 00:47:01.978572 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 950ba63893504c97b20e5b7fcd6df39f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36662
I20260927 00:47:01.978896 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 2196
I20260927 00:47:01.984544 2771 tablet_replica.cc:330] T 950ba63893504c97b20e5b7fcd6df39f P 3dd218cc6b694cd485ad6e25b5b73547: stopping tablet replica
I20260927 00:47:01.984814 2771 raft_consensus.cc:2229] T 950ba63893504c97b20e5b7fcd6df39f P 3dd218cc6b694cd485ad6e25b5b73547 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:47:01.985034 2770 tablet_replica.cc:330] T 950ba63893504c97b20e5b7fcd6df39f P 4d30b0c6d65d4dd1b23f9742e7ea1735: stopping tablet replica
I20260927 00:47:01.985072 2771 raft_consensus.cc:2258] T 950ba63893504c97b20e5b7fcd6df39f P 3dd218cc6b694cd485ad6e25b5b73547 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:01.985257 2770 raft_consensus.cc:2229] T 950ba63893504c97b20e5b7fcd6df39f P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:47:01.985637 2770 raft_consensus.cc:2258] T 950ba63893504c97b20e5b7fcd6df39f P 4d30b0c6d65d4dd1b23f9742e7ea1735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:01.986057 2771 ts_tablet_manager.cc:1905] T 950ba63893504c97b20e5b7fcd6df39f P 3dd218cc6b694cd485ad6e25b5b73547: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:01.986459 2770 ts_tablet_manager.cc:1905] T 950ba63893504c97b20e5b7fcd6df39f P 4d30b0c6d65d4dd1b23f9742e7ea1735: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 00:47:01.987458 2111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40547 (error 108)
I20260927 00:47:01.987931 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 2336
I20260927 00:47:01.990623 2773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:01.990087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40547 (local address 127.0.0.1:53482)
0927 00:47:01.990409 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:01.990498 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":113,"threads_started":1}
W20260927 00:47:01.990829 2111 catalog_manager.cc:4584] TS c84e3dfe390e42589ab1c1d7731889fc (127.0.98.1:40547): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 950ba63893504c97b20e5b7fcd6df39f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40547: connect: Connection refused (error 111)
I20260927 00:47:01.994212 2771 ts_tablet_manager.cc:1918] T 950ba63893504c97b20e5b7fcd6df39f P 3dd218cc6b694cd485ad6e25b5b73547: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 00:47:01.994338 2771 log.cc:1196] T 950ba63893504c97b20e5b7fcd6df39f P 3dd218cc6b694cd485ad6e25b5b73547: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790470006085796-392-0/minicluster-data/ts-2/wal/wals/950ba63893504c97b20e5b7fcd6df39f
I20260927 00:47:01.994654 2771 ts_tablet_manager.cc:1939] T 950ba63893504c97b20e5b7fcd6df39f P 3dd218cc6b694cd485ad6e25b5b73547: Deleting consensus metadata
I20260927 00:47:01.995864 2110 catalog_manager.cc:4841] TS 3dd218cc6b694cd485ad6e25b5b73547 (127.0.98.3:40181): tablet 950ba63893504c97b20e5b7fcd6df39f (table test-table [id=302bb82ed9a44a9a9705e0755c455b2e]) successfully deleted
I20260927 00:47:01.997753 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 2476
I20260927 00:47:01.998917 2773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:01.997989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43073 (local address 127.0.0.1:46622)
0927 00:47:01.998705 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:01.998782 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":591}
I20260927 00:47:02.005903 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 2092
2026-09-27T00:47:02Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6994 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:47:02 dist-test-slave-86rs krb5kdc[2776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:47:02 dist-test-slave-86rs krb5kdc[2776](info): set up 2 sockets
Sep 27 00:47:02 dist-test-slave-86rs krb5kdc[2776](info): commencing operation
krb5kdc: starting...
W20260927 00:47:03.088822 392 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:47:03 dist-test-slave-86rs krb5kdc[2776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:47:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:47:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:47:03.227707 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42919
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42919
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:03.371627 2789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:03.372556 2789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:03.372654 2789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:03.378226 2789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:47:03.378350 2789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:03.378373 2789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:03.378396 2789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:47:03.378414 2789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:47:03.385844 2789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:03.387495 2789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:03.389524 2789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:03.401803 2797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:03.401803 2794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:03.401803 2795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:03.402328 2789 server_base.cc:1034] running on GCE node
I20260927 00:47:03.403179 2789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:03.404839 2789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:03.406004 2789 hybrid_clock.cc:648] HybridClock initialized: now 1790470023405962 us; error 65 us; skew 500 ppm
Sep 27 00:47:03 dist-test-slave-86rs krb5kdc[2776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:03.409628 2789 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:47:03.410847 2789 webserver.cc:460] Webserver started at http://127.0.98.62:38685/ using document root <none> and password file <none>
I20260927 00:47:03.411578 2789 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:03.411648 2789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:03.411865 2789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:03.414729 2789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "fc80ab4729c142269716114780a1de7f"
format_stamp: "Formatted at 2026-09-27 00:47:03 on dist-test-slave-86rs"
server_key: "cf50bc6dc456af37080912768089561d"
server_key_iv: "d67fbe9b00ab848deb0e70b6a0113b4e"
server_key_version: "encryptionkey@0"
I20260927 00:47:03.415424 2789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "fc80ab4729c142269716114780a1de7f"
format_stamp: "Formatted at 2026-09-27 00:47:03 on dist-test-slave-86rs"
server_key: "cf50bc6dc456af37080912768089561d"
server_key_iv: "d67fbe9b00ab848deb0e70b6a0113b4e"
server_key_version: "encryptionkey@0"
I20260927 00:47:03.420960 2789 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 00:47:03.424371 2804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:03.426311 2789 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:47:03.426530 2789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "fc80ab4729c142269716114780a1de7f"
format_stamp: "Formatted at 2026-09-27 00:47:03 on dist-test-slave-86rs"
server_key: "cf50bc6dc456af37080912768089561d"
server_key_iv: "d67fbe9b00ab848deb0e70b6a0113b4e"
server_key_version: "encryptionkey@0"
I20260927 00:47:03.426733 2789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:03.439913 2789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:03.443857 2789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:03.444114 2789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:03.453116 2789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42919
I20260927 00:47:03.453141 2856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42919 every 8 connection(s)
I20260927 00:47:03.454603 2789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:47:03.457366 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 2789
I20260927 00:47:03.457521 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:47:03.457858 392 external_mini_cluster.cc:1428] Setting key e57a9647ee7c851d2223385caaa37c37
I20260927 00:47:03.459074 2857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:03.466708 2857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc80ab4729c142269716114780a1de7f: Bootstrap starting.
Sep 27 00:47:03 dist-test-slave-86rs krb5kdc[2776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:03.470166 2857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc80ab4729c142269716114780a1de7f: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:03.471261 2857 log.cc:824] T 00000000000000000000000000000000 P fc80ab4729c142269716114780a1de7f: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:03.474061 2857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc80ab4729c142269716114780a1de7f: No bootstrap required, opened a new log
I20260927 00:47:03.475148 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:03.459726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46138 (local address 127.0.98.62:42919)
0927 00:47:03.460398 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:03.460419 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:03.460468 (+ 49us) server_negotiation.cc:407] Connection header received
0927 00:47:03.461316 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:03.461342 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:03.461822 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:03.462292 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:03.463241 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:03.464444 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:03.465412 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:03.465991 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:03.466192 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:03.469251 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:03.469258 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:03.469278 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:03.469318 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:03.471489 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:03.472064 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:03.472071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:03.472078 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:03.472265 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:03.472597 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:03.472600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:03.472602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:03.473105 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:03.473345 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:03.473722 (+ 377us) server_negotiation.cc:299] Negotiation successful
0927 00:47:03.474107 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":200,"threads_started":1}
I20260927 00:47:03.478734 2857 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc80ab4729c142269716114780a1de7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc80ab4729c142269716114780a1de7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42919 } }
I20260927 00:47:03.479054 2857 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc80ab4729c142269716114780a1de7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:03.479138 2857 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc80ab4729c142269716114780a1de7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc80ab4729c142269716114780a1de7f, State: Initialized, Role: FOLLOWER
I20260927 00:47:03.479697 2857 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc80ab4729c142269716114780a1de7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc80ab4729c142269716114780a1de7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42919 } }
I20260927 00:47:03.479842 2857 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc80ab4729c142269716114780a1de7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:47:03.479938 2857 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc80ab4729c142269716114780a1de7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:47:03.480084 2857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc80ab4729c142269716114780a1de7f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:03.481573 2857 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc80ab4729c142269716114780a1de7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc80ab4729c142269716114780a1de7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42919 } }
I20260927 00:47:03.482148 2857 leader_election.cc:302] T 00000000000000000000000000000000 P fc80ab4729c142269716114780a1de7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc80ab4729c142269716114780a1de7f; no voters:
I20260927 00:47:03.482594 2857 leader_election.cc:288] T 00000000000000000000000000000000 P fc80ab4729c142269716114780a1de7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:47:03.482769 2862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc80ab4729c142269716114780a1de7f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:03.483042 2862 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc80ab4729c142269716114780a1de7f [term 1 LEADER]: Becoming Leader. State: Replica: fc80ab4729c142269716114780a1de7f, State: Running, Role: LEADER
I20260927 00:47:03.483505 2862 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc80ab4729c142269716114780a1de7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc80ab4729c142269716114780a1de7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42919 } }
I20260927 00:47:03.484349 2857 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc80ab4729c142269716114780a1de7f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:47:03.485519 2863 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc80ab4729c142269716114780a1de7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc80ab4729c142269716114780a1de7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc80ab4729c142269716114780a1de7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42919 } } }
I20260927 00:47:03.485658 2863 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc80ab4729c142269716114780a1de7f [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:03.486002 2864 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc80ab4729c142269716114780a1de7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc80ab4729c142269716114780a1de7f. Latest consensus state: current_term: 1 leader_uuid: "fc80ab4729c142269716114780a1de7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc80ab4729c142269716114780a1de7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42919 } } }
I20260927 00:47:03.486096 2864 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc80ab4729c142269716114780a1de7f [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:03.486443 2871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:47:03.490245 2871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:47:03.497800 2871 catalog_manager.cc:1347] Generated new cluster ID: 6c0fc7c5295f44a79ec4e4a02fce0842
I20260927 00:47:03.497932 2871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:47:03.522250 2871 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:47:03.523734 2871 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:47:03.533893 2871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc80ab4729c142269716114780a1de7f: Generated new TSK 0
I20260927 00:47:03.535290 2871 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:47:03.593319 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42919
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:03.758148 2883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:03.758587 2883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:03.758641 2883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:03.763912 2883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:03.764030 2883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:03.764166 2883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:47:03.770874 2883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:03.772578 2883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:03.774008 2883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:03.787251 2891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:03.787247 2888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:03.787254 2889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:03.787817 2883 server_base.cc:1034] running on GCE node
I20260927 00:47:03.788388 2883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:03.789184 2883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:03.790393 2883 hybrid_clock.cc:648] HybridClock initialized: now 1790470023790371 us; error 35 us; skew 500 ppm
Sep 27 00:47:03 dist-test-slave-86rs krb5kdc[2776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:03.793903 2883 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:47:03.795082 2883 webserver.cc:460] Webserver started at http://127.0.98.1:32993/ using document root <none> and password file <none>
I20260927 00:47:03.795768 2883 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:03.795833 2883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:03.796049 2883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:03.798878 2883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/data/instance:
uuid: "2ab5b05362054579ae865793a726d552"
format_stamp: "Formatted at 2026-09-27 00:47:03 on dist-test-slave-86rs"
server_key: "51cc28f4453333712a039fd6e53a0a18"
server_key_iv: "56cf9256950b616b865b9f6eb461bbec"
server_key_version: "encryptionkey@0"
I20260927 00:47:03.799518 2883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2ab5b05362054579ae865793a726d552"
format_stamp: "Formatted at 2026-09-27 00:47:03 on dist-test-slave-86rs"
server_key: "51cc28f4453333712a039fd6e53a0a18"
server_key_iv: "56cf9256950b616b865b9f6eb461bbec"
server_key_version: "encryptionkey@0"
I20260927 00:47:03.804703 2883 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 00:47:03.808164 2898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:03.810073 2883 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:47:03.810292 2883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "2ab5b05362054579ae865793a726d552"
format_stamp: "Formatted at 2026-09-27 00:47:03 on dist-test-slave-86rs"
server_key: "51cc28f4453333712a039fd6e53a0a18"
server_key_iv: "56cf9256950b616b865b9f6eb461bbec"
server_key_version: "encryptionkey@0"
I20260927 00:47:03.810456 2883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:03.834491 2883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:03.838785 2883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:03.838995 2883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:03.839874 2883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:03.841024 2883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:03.841078 2883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:03.841126 2883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:03.841142 2883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:03.852293 2883 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40685
I20260927 00:47:03.852336 3011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40685 every 8 connection(s)
I20260927 00:47:03.853592 2883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:47:03.854111 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 2883
I20260927 00:47:03.854249 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:47:03.854629 392 external_mini_cluster.cc:1428] Setting key 7be602de6f19195b0029b5fccf102032
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 27 00:47:03 dist-test-slave-86rs krb5kdc[2776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:03.876204 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:03.856522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60195 (local address 127.0.98.62:42919)
0927 00:47:03.856692 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:03.856698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:03.858208 (+ 1510us) server_negotiation.cc:407] Connection header received
0927 00:47:03.860432 (+ 2224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:03.860436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:03.860494 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:03.860572 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:03.862686 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:03.863563 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:03.865048 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:03.865459 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:03.865596 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:03.871310 (+ 5714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:03.871314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:03.871316 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:03.871350 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:03.873354 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:03.874040 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:03.874047 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:03.874052 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:03.874172 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:03.874572 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:03.874578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:03.874583 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:03.874885 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:03.875000 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:03.875716 (+ 716us) server_negotiation.cc:299] Negotiation successful
0927 00:47:03.875907 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 00:47:03.877266 3014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:03.857302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42919 (local address 127.0.98.1:60195)
0927 00:47:03.857984 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:03.858051 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:03.860090 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:03.860835 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:03.860850 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:03.861511 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:03.862460 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:03.862491 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:03.863742 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:03.863745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:03.864905 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:03.864915 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:03.865668 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:03.865672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:03.865871 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:03.866713 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:03.866734 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:03.871050 (+ 4316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:03.873537 (+ 2487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:03.873545 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:03.873566 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:03.873901 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:03.874283 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:03.874285 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:03.874287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:03.874460 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:03.875021 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:03.875028 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:03.875367 (+ 339us) client_negotiation.cc:769] Sending connection context
0927 00:47:03.875597 (+ 230us) client_negotiation.cc:240] Negotiation successful
0927 00:47:03.875968 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":203,"threads_started":1}
I20260927 00:47:03.878989 3012 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42919
I20260927 00:47:03.879338 3012 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:03.880081 3012 heartbeater.cc:511] Master 127.0.98.62:42919 requested a full tablet report, sending...
I20260927 00:47:03.882608 2821 ts_manager.cc:194] Registered new tserver with Master: 2ab5b05362054579ae865793a726d552 (127.0.98.1:40685)
I20260927 00:47:03.884086 2821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:47:03.889506 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42919
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:04.031387 3017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:04.031867 3017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:04.031956 3017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:04.036617 3017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:04.036753 3017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:04.036863 3017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:47:04.043197 3017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:04.044854 3017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:04.046296 3017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:04.058585 3025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:04.058655 3022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:04.058787 3017 server_base.cc:1034] running on GCE node
W20260927 00:47:04.058643 3023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:04.059465 3017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:04.060401 3017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:04.061620 3017 hybrid_clock.cc:648] HybridClock initialized: now 1790470024061601 us; error 43 us; skew 500 ppm
Sep 27 00:47:04 dist-test-slave-86rs krb5kdc[2776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:04.065366 3017 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:47:04.066598 3017 webserver.cc:460] Webserver started at http://127.0.98.2:36281/ using document root <none> and password file <none>
I20260927 00:47:04.067386 3017 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:04.067461 3017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:04.067683 3017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:04.070456 3017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/data/instance:
uuid: "a046aaa1093142e599c4b8becfcb332f"
format_stamp: "Formatted at 2026-09-27 00:47:04 on dist-test-slave-86rs"
server_key: "7a297cbe3a49e909725ae4ffe374cd46"
server_key_iv: "e45310080d30087bc1846a813f6e5398"
server_key_version: "encryptionkey@0"
I20260927 00:47:04.071208 3017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a046aaa1093142e599c4b8becfcb332f"
format_stamp: "Formatted at 2026-09-27 00:47:04 on dist-test-slave-86rs"
server_key: "7a297cbe3a49e909725ae4ffe374cd46"
server_key_iv: "e45310080d30087bc1846a813f6e5398"
server_key_version: "encryptionkey@0"
I20260927 00:47:04.076574 3017 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 00:47:04.079959 3032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:04.081931 3017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:47:04.082077 3017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "a046aaa1093142e599c4b8becfcb332f"
format_stamp: "Formatted at 2026-09-27 00:47:04 on dist-test-slave-86rs"
server_key: "7a297cbe3a49e909725ae4ffe374cd46"
server_key_iv: "e45310080d30087bc1846a813f6e5398"
server_key_version: "encryptionkey@0"
I20260927 00:47:04.082199 3017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:04.094954 3017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:04.099645 3017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:04.099867 3017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:04.100802 3017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:04.101977 3017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:04.102041 3017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:04.102098 3017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:04.102116 3017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:04.115260 3017 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40629
I20260927 00:47:04.115327 3145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40629 every 8 connection(s)
I20260927 00:47:04.116653 3017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:47:04.118474 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 3017
I20260927 00:47:04.118626 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:47:04.118939 392 external_mini_cluster.cc:1428] Setting key 500356941063c3235870ced5c95ee76c
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 27 00:47:04 dist-test-slave-86rs krb5kdc[2776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:47:04.137389 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:04.119467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46621 (local address 127.0.98.62:42919)
0927 00:47:04.119644 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:04.119650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:04.121278 (+ 1628us) server_negotiation.cc:407] Connection header received
0927 00:47:04.123776 (+ 2498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:04.123780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:04.123856 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:04.123948 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:04.125696 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:04.126304 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:04.127761 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:04.128017 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:04.128146 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:04.131880 (+ 3734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:04.131887 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:04.131891 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:04.131975 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:04.134194 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:04.134956 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:04.134961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:04.134964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:04.135068 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:04.135526 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:04.135530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:04.135532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:04.135892 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:04.136024 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:04.136913 (+ 889us) server_negotiation.cc:299] Negotiation successful
0927 00:47:04.137107 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:47:04.138821 3148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:04.120289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42919 (local address 127.0.98.2:46621)
0927 00:47:04.121059 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:04.121134 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:04.123461 (+ 2327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:04.124216 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:04.124276 (+ 60us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:04.124799 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:04.125497 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:04.125515 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:04.126473 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:04.126477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:04.127595 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:04.127607 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:04.128269 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:04.128275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:04.128502 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:04.129488 (+ 986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:04.129512 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:04.131580 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:04.134375 (+ 2795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:04.134386 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:04.134422 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:04.134812 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:04.135192 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:04.135196 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:04.135198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:04.135404 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:04.136081 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:04.136089 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:04.136546 (+ 457us) client_negotiation.cc:769] Sending connection context
0927 00:47:04.136802 (+ 256us) client_negotiation.cc:240] Negotiation successful
0927 00:47:04.137236 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":212,"threads_started":1}
I20260927 00:47:04.140627 3146 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42919
I20260927 00:47:04.141021 3146 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:04.141778 3146 heartbeater.cc:511] Master 127.0.98.62:42919 requested a full tablet report, sending...
I20260927 00:47:04.143291 2821 ts_manager.cc:194] Registered new tserver with Master: a046aaa1093142e599c4b8becfcb332f (127.0.98.2:40629)
I20260927 00:47:04.144027 2821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:47:04.158123 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42919
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:04.306370 3151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:04.306826 3151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:04.306917 3151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:04.312126 3151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:04.312314 3151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:04.312439 3151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:47:04.320096 3151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:04.321898 3151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:04.323423 3151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:04.336134 3159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:04.336162 3156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:04.336248 3151 server_base.cc:1034] running on GCE node
W20260927 00:47:04.336138 3157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:04.337019 3151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:04.337879 3151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:04.339095 3151 hybrid_clock.cc:648] HybridClock initialized: now 1790470024339060 us; error 46 us; skew 500 ppm
Sep 27 00:47:04 dist-test-slave-86rs krb5kdc[2776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:04.342517 3151 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:47:04.343683 3151 webserver.cc:460] Webserver started at http://127.0.98.3:32985/ using document root <none> and password file <none>
I20260927 00:47:04.344484 3151 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:04.344599 3151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:04.344866 3151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:04.347540 3151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/data/instance:
uuid: "4c7cadf3c1344b559e834fc04ab72d70"
format_stamp: "Formatted at 2026-09-27 00:47:04 on dist-test-slave-86rs"
server_key: "ebac9ea0cdacd0fc1610d2224041dcc7"
server_key_iv: "a2807e8927d87f267ddc5d16e4100227"
server_key_version: "encryptionkey@0"
I20260927 00:47:04.348289 3151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4c7cadf3c1344b559e834fc04ab72d70"
format_stamp: "Formatted at 2026-09-27 00:47:04 on dist-test-slave-86rs"
server_key: "ebac9ea0cdacd0fc1610d2224041dcc7"
server_key_iv: "a2807e8927d87f267ddc5d16e4100227"
server_key_version: "encryptionkey@0"
I20260927 00:47:04.353114 3151 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260927 00:47:04.356349 3166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:04.357980 3151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:47:04.358139 3151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "4c7cadf3c1344b559e834fc04ab72d70"
format_stamp: "Formatted at 2026-09-27 00:47:04 on dist-test-slave-86rs"
server_key: "ebac9ea0cdacd0fc1610d2224041dcc7"
server_key_iv: "a2807e8927d87f267ddc5d16e4100227"
server_key_version: "encryptionkey@0"
I20260927 00:47:04.358276 3151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:04.381078 3151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:04.385692 3151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:04.385999 3151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:04.386979 3151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:04.388329 3151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:04.388402 3151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:04.388484 3151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:04.388525 3151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:04.399287 3151 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46655
I20260927 00:47:04.399312 3279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46655 every 8 connection(s)
I20260927 00:47:04.400594 3151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:47:04.408195 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 3151
I20260927 00:47:04.408427 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:47:04.408789 392 external_mini_cluster.cc:1428] Setting key c186b48ae786fad63c3af8086a6bf6ed
Sep 27 00:47:04 dist-test-slave-86rs krb5kdc[2776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:04.417596 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:04.403081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59237 (local address 127.0.98.62:42919)
0927 00:47:04.403256 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:04.403260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:04.404581 (+ 1321us) server_negotiation.cc:407] Connection header received
0927 00:47:04.406308 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:04.406313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:04.406384 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:04.406504 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:04.407987 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:04.408742 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:04.410172 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:04.410406 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:04.410527 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:04.413588 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:04.413592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:04.413594 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:04.413625 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:04.415216 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:04.415810 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:04.415815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:04.415817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:04.415913 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:04.416336 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:04.416342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:04.416346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:04.416604 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:04.416704 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:04.417306 (+ 602us) server_negotiation.cc:299] Negotiation successful
0927 00:47:04.417443 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 00:47:04.418780 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:04.403686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42919 (local address 127.0.98.3:59237)
0927 00:47:04.404395 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:04.404450 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:04.406047 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:04.406736 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:04.406746 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:04.407204 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:04.407827 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:04.407842 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:04.408895 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:04.408903 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:04.410046 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:04.410055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:04.410577 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:04.410582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:04.410786 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:04.411506 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:04.411526 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:04.413357 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:04.415351 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:04.415358 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:04.415380 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:04.415696 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:04.416012 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:04.416016 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:04.416018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:04.416186 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:04.416710 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:04.416727 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:04.417043 (+ 316us) client_negotiation.cc:769] Sending connection context
0927 00:47:04.417245 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 00:47:04.417601 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":200,"threads_started":1}
I20260927 00:47:04.420305 3280 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42919
I20260927 00:47:04.420601 3280 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:04.421234 3280 heartbeater.cc:511] Master 127.0.98.62:42919 requested a full tablet report, sending...
I20260927 00:47:04.422523 2821 ts_manager.cc:194] Registered new tserver with Master: 4c7cadf3c1344b559e834fc04ab72d70 (127.0.98.3:46655)
I20260927 00:47:04.423218 2821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59237
I20260927 00:47:04.423779 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 00:47:04.440158 2860 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:04.438294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46140 (local address 127.0.98.62:42919)
0927 00:47:04.438474 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:04.438480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:04.438624 (+ 144us) server_negotiation.cc:407] Connection header received
0927 00:47:04.438843 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:04.438845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:04.438900 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:04.438997 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:04.439953 (+ 956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46140: BlockingRecv error: recv got EOF from 127.0.0.1:46140 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:47:04.441392 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 2883
I20260927 00:47:04.448415 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 3017
I20260927 00:47:04.455596 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 3151
I20260927 00:47:04.463914 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 2789
2026-09-27T00:47:04Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2456 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:47:04 dist-test-slave-86rs krb5kdc[3292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:47:04 dist-test-slave-86rs krb5kdc[3292](info): set up 2 sockets
Sep 27 00:47:04 dist-test-slave-86rs krb5kdc[3292](info): commencing operation
krb5kdc: starting...
W20260927 00:47:05.548772 392 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260927 00:47:05.702428 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.153s user 0.002s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260927 00:47:05.898909 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.196s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260927 00:47:06.139400 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.240s user 0.000s sys 0.003s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470006085796-392-0/krb5kdc/test-user.keytab.
W20260927 00:47:06.682957 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.543s user 0.000s sys 0.002s
Sep 27 00:47:06 dist-test-slave-86rs krb5kdc[3292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:47:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:47:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260927 00:47:07.068696 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.356s user 0.000s sys 0.004s
I20260927 00:47:07.069554 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45313
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:07.213546 3305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:07.213973 3305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:07.214025 3305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:07.219398 3305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:47:07.219493 3305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:07.219515 3305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:07.219535 3305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:47:07.219552 3305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:47:07.225808 3305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:07.227392 3305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:07.228909 3305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:07.238991 3313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:07.239022 3310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:07.239006 3311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:07.239603 3305 server_base.cc:1034] running on GCE node
I20260927 00:47:07.240406 3305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:07.241905 3305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:07.243077 3305 hybrid_clock.cc:648] HybridClock initialized: now 1790470027243060 us; error 37 us; skew 500 ppm
Sep 27 00:47:07 dist-test-slave-86rs krb5kdc[3292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:07.246474 3305 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:47:07.247704 3305 webserver.cc:460] Webserver started at http://127.0.98.62:37187/ using document root <none> and password file <none>
I20260927 00:47:07.248554 3305 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:07.248634 3305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:07.248905 3305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:07.251617 3305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "8dcd3ecced0b4485931a7b021d43df9f"
format_stamp: "Formatted at 2026-09-27 00:47:07 on dist-test-slave-86rs"
server_key: "f438be2113d644d77900ae75615efe8e"
server_key_iv: "09b1c75bc4c9786133dfcc1dbde81726"
server_key_version: "encryptionkey@0"
I20260927 00:47:07.252386 3305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "8dcd3ecced0b4485931a7b021d43df9f"
format_stamp: "Formatted at 2026-09-27 00:47:07 on dist-test-slave-86rs"
server_key: "f438be2113d644d77900ae75615efe8e"
server_key_iv: "09b1c75bc4c9786133dfcc1dbde81726"
server_key_version: "encryptionkey@0"
I20260927 00:47:07.257242 3305 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260927 00:47:07.260579 3320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:07.262316 3305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:47:07.262517 3305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "8dcd3ecced0b4485931a7b021d43df9f"
format_stamp: "Formatted at 2026-09-27 00:47:07 on dist-test-slave-86rs"
server_key: "f438be2113d644d77900ae75615efe8e"
server_key_iv: "09b1c75bc4c9786133dfcc1dbde81726"
server_key_version: "encryptionkey@0"
I20260927 00:47:07.262681 3305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:07.283151 3305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:07.287374 3305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:07.287644 3305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:07.296650 3305 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45313
I20260927 00:47:07.296976 3372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45313 every 8 connection(s)
I20260927 00:47:07.297962 3305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:47:07.299755 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 3305
I20260927 00:47:07.299921 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:47:07.300269 392 external_mini_cluster.cc:1428] Setting key de12940b39fc6efd532a845f4b74d4a4
I20260927 00:47:07.302189 3373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:07.311498 3373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dcd3ecced0b4485931a7b021d43df9f: Bootstrap starting.
Sep 27 00:47:07 dist-test-slave-86rs krb5kdc[3292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:07.315376 3373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8dcd3ecced0b4485931a7b021d43df9f: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:07.316581 3373 log.cc:824] T 00000000000000000000000000000000 P 8dcd3ecced0b4485931a7b021d43df9f: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:07.318739 3376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:07.302036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53100 (local address 127.0.98.62:45313)
0927 00:47:07.302598 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:07.302631 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:07.302677 (+ 46us) server_negotiation.cc:407] Connection header received
0927 00:47:07.303602 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:07.303639 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:07.304326 (+ 687us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:07.304816 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:07.305876 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:07.307505 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:07.308632 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:07.309441 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:07.309641 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:07.312780 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:07.312788 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:07.312809 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:07.312849 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:07.314968 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:07.315586 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:07.315593 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:07.315601 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:07.315781 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:07.316073 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:07.316076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:07.316078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:07.316782 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:07.316993 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:07.317329 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 00:47:07.317729 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":226,"threads_started":1}
I20260927 00:47:07.319546 3373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dcd3ecced0b4485931a7b021d43df9f: No bootstrap required, opened a new log
I20260927 00:47:07.322907 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:47:07.324518 3373 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8dcd3ecced0b4485931a7b021d43df9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dcd3ecced0b4485931a7b021d43df9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45313 } }
I20260927 00:47:07.324860 3373 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8dcd3ecced0b4485931a7b021d43df9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:07.324961 3373 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8dcd3ecced0b4485931a7b021d43df9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dcd3ecced0b4485931a7b021d43df9f, State: Initialized, Role: FOLLOWER
I20260927 00:47:07.325716 3373 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8dcd3ecced0b4485931a7b021d43df9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dcd3ecced0b4485931a7b021d43df9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45313 } }
I20260927 00:47:07.325932 3373 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8dcd3ecced0b4485931a7b021d43df9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:47:07.326016 3373 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8dcd3ecced0b4485931a7b021d43df9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:47:07.326159 3373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8dcd3ecced0b4485931a7b021d43df9f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:07.327557 3373 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8dcd3ecced0b4485931a7b021d43df9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dcd3ecced0b4485931a7b021d43df9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45313 } }
I20260927 00:47:07.328027 3373 leader_election.cc:302] T 00000000000000000000000000000000 P 8dcd3ecced0b4485931a7b021d43df9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8dcd3ecced0b4485931a7b021d43df9f; no voters:
I20260927 00:47:07.328510 3373 leader_election.cc:288] T 00000000000000000000000000000000 P 8dcd3ecced0b4485931a7b021d43df9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:47:07.328609 3379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8dcd3ecced0b4485931a7b021d43df9f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:07.328850 3379 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8dcd3ecced0b4485931a7b021d43df9f [term 1 LEADER]: Becoming Leader. State: Replica: 8dcd3ecced0b4485931a7b021d43df9f, State: Running, Role: LEADER
I20260927 00:47:07.329293 3379 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8dcd3ecced0b4485931a7b021d43df9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dcd3ecced0b4485931a7b021d43df9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45313 } }
I20260927 00:47:07.330276 3373 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8dcd3ecced0b4485931a7b021d43df9f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:47:07.330979 3381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dcd3ecced0b4485931a7b021d43df9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dcd3ecced0b4485931a7b021d43df9f. Latest consensus state: current_term: 1 leader_uuid: "8dcd3ecced0b4485931a7b021d43df9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dcd3ecced0b4485931a7b021d43df9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45313 } } }
I20260927 00:47:07.331144 3381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dcd3ecced0b4485931a7b021d43df9f [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:07.331427 3380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dcd3ecced0b4485931a7b021d43df9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8dcd3ecced0b4485931a7b021d43df9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dcd3ecced0b4485931a7b021d43df9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45313 } } }
I20260927 00:47:07.331650 3380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dcd3ecced0b4485931a7b021d43df9f [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:07.332114 3386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:47:07.336550 3386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:47:07.345738 3386 catalog_manager.cc:1347] Generated new cluster ID: 344c0eda902843cea3298ce1abbc0db9
I20260927 00:47:07.345872 3386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:47:07.346997 3376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:07.340602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53110 (local address 127.0.98.62:45313)
0927 00:47:07.340778 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:07.340786 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:07.340804 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:47:07.340872 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:07.340875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:07.340926 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:07.341036 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:07.341718 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:07.342371 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:07.343633 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:07.344034 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:07.344174 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:07.345656 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:07.345660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:07.345664 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:47:07.346392 (+ 728us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:07.346444 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:47:07.346468 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:07.346576 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:07.346731 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 00:47:07.346821 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 00:47:07.389626 3386 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:47:07.391201 3386 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:47:07.397207 3386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8dcd3ecced0b4485931a7b021d43df9f: Generated new TSK 0
I20260927 00:47:07.398308 3386 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:47:07.404985 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 3305
2026-09-27T00:47:07Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2937 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:47:07 dist-test-slave-86rs krb5kdc[3406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:47:07 dist-test-slave-86rs krb5kdc[3406](info): set up 2 sockets
Sep 27 00:47:07 dist-test-slave-86rs krb5kdc[3406](info): commencing operation
krb5kdc: starting...
W20260927 00:47:08.452857 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:47:08 dist-test-slave-86rs krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:47:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:47:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:47:08.591681 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39651
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:08.735766 3419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:08.736213 3419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:08.736402 3419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:08.741137 3419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:47:08.741251 3419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:08.741276 3419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:08.741298 3419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:47:08.741318 3419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:47:08.748337 3419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:08.750001 3419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:08.751521 3419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:08.762789 3425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:08.762799 3424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:08.762787 3427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:08.763238 3419 server_base.cc:1034] running on GCE node
I20260927 00:47:08.764026 3419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:08.765666 3419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:08.766924 3419 hybrid_clock.cc:648] HybridClock initialized: now 1790470028766883 us; error 67 us; skew 500 ppm
Sep 27 00:47:08 dist-test-slave-86rs krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:08.770961 3419 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:47:08.772327 3419 webserver.cc:460] Webserver started at http://127.0.98.62:33791/ using document root <none> and password file <none>
I20260927 00:47:08.773102 3419 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:08.773214 3419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:08.773502 3419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:08.776279 3419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "150bb6cf76d8465fb5794ff7ac78c406"
format_stamp: "Formatted at 2026-09-27 00:47:08 on dist-test-slave-86rs"
server_key: "1fa397a5fe0129ccee5d8bc81e6c364f"
server_key_iv: "f66f309395cd256b16622a40855f7716"
server_key_version: "encryptionkey@0"
I20260927 00:47:08.777032 3419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "150bb6cf76d8465fb5794ff7ac78c406"
format_stamp: "Formatted at 2026-09-27 00:47:08 on dist-test-slave-86rs"
server_key: "1fa397a5fe0129ccee5d8bc81e6c364f"
server_key_iv: "f66f309395cd256b16622a40855f7716"
server_key_version: "encryptionkey@0"
I20260927 00:47:08.782433 3419 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 00:47:08.785818 3434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:08.787601 3419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:47:08.787789 3419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "150bb6cf76d8465fb5794ff7ac78c406"
format_stamp: "Formatted at 2026-09-27 00:47:08 on dist-test-slave-86rs"
server_key: "1fa397a5fe0129ccee5d8bc81e6c364f"
server_key_iv: "f66f309395cd256b16622a40855f7716"
server_key_version: "encryptionkey@0"
I20260927 00:47:08.787945 3419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:08.814476 3419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:08.818819 3419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:08.819075 3419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:08.827898 3419 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39651
I20260927 00:47:08.828296 3486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39651 every 8 connection(s)
I20260927 00:47:08.829263 3419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:47:08.831071 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 3419
I20260927 00:47:08.831228 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:47:08.831547 392 external_mini_cluster.cc:1428] Setting key 3589bd8fd42b03e6c477a1e234461c65
I20260927 00:47:08.833511 3487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:08.842502 3487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 150bb6cf76d8465fb5794ff7ac78c406: Bootstrap starting.
Sep 27 00:47:08 dist-test-slave-86rs krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:08.846313 3487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 150bb6cf76d8465fb5794ff7ac78c406: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:08.847486 3487 log.cc:824] T 00000000000000000000000000000000 P 150bb6cf76d8465fb5794ff7ac78c406: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:08.849715 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:08.833414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33436 (local address 127.0.98.62:39651)
0927 00:47:08.833963 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:08.833984 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:08.834032 (+ 48us) server_negotiation.cc:407] Connection header received
0927 00:47:08.834834 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:08.834857 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:08.835313 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:08.835697 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:08.836837 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:08.837995 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:08.839396 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:08.839978 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:08.840180 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:08.843610 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:08.843618 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:08.843639 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:08.843681 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:08.845849 (+ 2168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:08.846376 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:08.846387 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:08.846395 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:08.846558 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:08.846857 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:08.846860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:08.846862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:08.847512 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:08.847737 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:08.848082 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 00:47:08.848513 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":200,"threads_started":1}
I20260927 00:47:08.850404 3487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 150bb6cf76d8465fb5794ff7ac78c406: No bootstrap required, opened a new log
I20260927 00:47:08.854830 3487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 150bb6cf76d8465fb5794ff7ac78c406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150bb6cf76d8465fb5794ff7ac78c406" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39651 } }
I20260927 00:47:08.855196 3487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 150bb6cf76d8465fb5794ff7ac78c406 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:08.855290 3487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 150bb6cf76d8465fb5794ff7ac78c406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 150bb6cf76d8465fb5794ff7ac78c406, State: Initialized, Role: FOLLOWER
I20260927 00:47:08.855899 3487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 150bb6cf76d8465fb5794ff7ac78c406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150bb6cf76d8465fb5794ff7ac78c406" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39651 } }
I20260927 00:47:08.856084 3487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 150bb6cf76d8465fb5794ff7ac78c406 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:47:08.856185 3487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 150bb6cf76d8465fb5794ff7ac78c406 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:47:08.856334 3487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 150bb6cf76d8465fb5794ff7ac78c406 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:08.857542 3487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 150bb6cf76d8465fb5794ff7ac78c406 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150bb6cf76d8465fb5794ff7ac78c406" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39651 } }
I20260927 00:47:08.857995 3487 leader_election.cc:302] T 00000000000000000000000000000000 P 150bb6cf76d8465fb5794ff7ac78c406 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 150bb6cf76d8465fb5794ff7ac78c406; no voters:
I20260927 00:47:08.858469 3487 leader_election.cc:288] T 00000000000000000000000000000000 P 150bb6cf76d8465fb5794ff7ac78c406 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:47:08.858598 3492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 150bb6cf76d8465fb5794ff7ac78c406 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:08.858897 3492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 150bb6cf76d8465fb5794ff7ac78c406 [term 1 LEADER]: Becoming Leader. State: Replica: 150bb6cf76d8465fb5794ff7ac78c406, State: Running, Role: LEADER
I20260927 00:47:08.859365 3492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 150bb6cf76d8465fb5794ff7ac78c406 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150bb6cf76d8465fb5794ff7ac78c406" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39651 } }
I20260927 00:47:08.860136 3487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 150bb6cf76d8465fb5794ff7ac78c406 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:47:08.861483 3494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 150bb6cf76d8465fb5794ff7ac78c406 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 150bb6cf76d8465fb5794ff7ac78c406. Latest consensus state: current_term: 1 leader_uuid: "150bb6cf76d8465fb5794ff7ac78c406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150bb6cf76d8465fb5794ff7ac78c406" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39651 } } }
I20260927 00:47:08.861634 3494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 150bb6cf76d8465fb5794ff7ac78c406 [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:08.862051 3501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:47:08.862008 3493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 150bb6cf76d8465fb5794ff7ac78c406 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "150bb6cf76d8465fb5794ff7ac78c406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150bb6cf76d8465fb5794ff7ac78c406" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39651 } } }
I20260927 00:47:08.862165 3493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 150bb6cf76d8465fb5794ff7ac78c406 [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:08.865868 3501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:47:08.873026 3501 catalog_manager.cc:1347] Generated new cluster ID: 58f59c57a3fc42b5a30ef0cc3e67f913
I20260927 00:47:08.873135 3501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:47:08.892158 3501 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:47:08.893721 3501 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:47:08.906590 3501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 150bb6cf76d8465fb5794ff7ac78c406: Generated new TSK 0
I20260927 00:47:08.907688 3501 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:47:08.943722 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39651
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:09.089918 3513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:09.090394 3513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:09.090493 3513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:09.096874 3513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:09.097041 3513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:09.097185 3513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:47:09.104542 3513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:09.106354 3513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:09.107972 3513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:09.122300 3518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:09.122282 3521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:09.122282 3519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:09.122723 3513 server_base.cc:1034] running on GCE node
I20260927 00:47:09.123478 3513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:09.124500 3513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:09.125703 3513 hybrid_clock.cc:648] HybridClock initialized: now 1790470029125653 us; error 69 us; skew 500 ppm
Sep 27 00:47:09 dist-test-slave-86rs krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:09.130249 3513 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:47:09.131646 3513 webserver.cc:460] Webserver started at http://127.0.98.1:35687/ using document root <none> and password file <none>
I20260927 00:47:09.132601 3513 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:09.132725 3513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:09.133010 3513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:09.135752 3513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/data/instance:
uuid: "31a8e24689c24d2a9619d00b5b7b3a48"
format_stamp: "Formatted at 2026-09-27 00:47:09 on dist-test-slave-86rs"
server_key: "2a4f003adc78c280d6c4aebb2be5dab4"
server_key_iv: "5bc335c84d56d3b6b36601e036c8b20e"
server_key_version: "encryptionkey@0"
I20260927 00:47:09.136528 3513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/wal/instance:
uuid: "31a8e24689c24d2a9619d00b5b7b3a48"
format_stamp: "Formatted at 2026-09-27 00:47:09 on dist-test-slave-86rs"
server_key: "2a4f003adc78c280d6c4aebb2be5dab4"
server_key_iv: "5bc335c84d56d3b6b36601e036c8b20e"
server_key_version: "encryptionkey@0"
I20260927 00:47:09.141476 3513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 00:47:09.145037 3528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:09.146799 3513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:47:09.146986 3513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "31a8e24689c24d2a9619d00b5b7b3a48"
format_stamp: "Formatted at 2026-09-27 00:47:09 on dist-test-slave-86rs"
server_key: "2a4f003adc78c280d6c4aebb2be5dab4"
server_key_iv: "5bc335c84d56d3b6b36601e036c8b20e"
server_key_version: "encryptionkey@0"
I20260927 00:47:09.147177 3513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:09.164795 3513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:09.169152 3513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:09.169423 3513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:09.170262 3513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:09.171386 3513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:09.171466 3513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:09.171547 3513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:09.171595 3513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:09.182753 3513 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36103
I20260927 00:47:09.182775 3641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36103 every 8 connection(s)
I20260927 00:47:09.184104 3513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:47:09.193980 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 3513
I20260927 00:47:09.194214 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:47:09.194643 392 external_mini_cluster.cc:1428] Setting key 00652a10f652e8aafcee849101cff09e
Sep 27 00:47:09 dist-test-slave-86rs krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:09.203925 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:09.186765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56353 (local address 127.0.98.62:39651)
0927 00:47:09.186954 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:09.186958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:09.188154 (+ 1196us) server_negotiation.cc:407] Connection header received
0927 00:47:09.189955 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:09.189959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:09.190054 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:09.190161 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:09.191783 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:09.192575 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:09.194110 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:09.194506 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:09.194648 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:09.198713 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:09.198721 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:09.198728 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:09.198773 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:09.200888 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:09.201576 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:09.201583 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:09.201588 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:09.201705 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:09.202122 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:09.202129 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:09.202134 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:09.202487 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:09.202624 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:09.203415 (+ 791us) server_negotiation.cc:299] Negotiation successful
0927 00:47:09.203622 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 00:47:09.205058 3644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:09.187320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39651 (local address 127.0.98.1:56353)
0927 00:47:09.187938 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:09.187994 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:09.189622 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:09.190406 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:09.190416 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:09.190929 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:09.191595 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:09.191611 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:09.192744 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:09.192748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:09.193951 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:09.193963 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:09.194812 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:09.194819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:09.195102 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:09.195985 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:09.196007 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:09.198421 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:09.201046 (+ 2625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:09.201053 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:09.201077 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:09.201450 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:09.201810 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:09.201812 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:09.201814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:09.202009 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:09.202638 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:09.202646 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:09.203069 (+ 423us) client_negotiation.cc:769] Sending connection context
0927 00:47:09.203309 (+ 240us) client_negotiation.cc:240] Negotiation successful
0927 00:47:09.203704 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":156,"threads_started":1}
I20260927 00:47:09.206852 3642 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39651
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:47:09.207235 3642 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:09.207957 3642 heartbeater.cc:511] Master 127.0.98.62:39651 requested a full tablet report, sending...
I20260927 00:47:09.210199 3451 ts_manager.cc:194] Registered new tserver with Master: 31a8e24689c24d2a9619d00b5b7b3a48 (127.0.98.1:36103)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:47:09.212357 3451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:47:09.232769 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39651
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:09.375456 3647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:09.375823 3647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:09.375903 3647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:09.381019 3647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:09.381117 3647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:09.381229 3647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:47:09.387444 3647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:09.389062 3647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:09.390533 3647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:09.403275 3652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:09.403319 3653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:09.403509 3647 server_base.cc:1034] running on GCE node
W20260927 00:47:09.403405 3655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:09.404258 3647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:09.405186 3647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:09.406412 3647 hybrid_clock.cc:648] HybridClock initialized: now 1790470029406391 us; error 37 us; skew 500 ppm
Sep 27 00:47:09 dist-test-slave-86rs krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:09.410017 3647 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:47:09.411201 3647 webserver.cc:460] Webserver started at http://127.0.98.2:44545/ using document root <none> and password file <none>
I20260927 00:47:09.411926 3647 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:09.411999 3647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:09.412281 3647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:09.415343 3647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/data/instance:
uuid: "c8cffc76435d41958287ed0e34cab65f"
format_stamp: "Formatted at 2026-09-27 00:47:09 on dist-test-slave-86rs"
server_key: "ea67521ce77100ec086adc9880cb1963"
server_key_iv: "4b5972a026f7e6ab9043813ed53554fd"
server_key_version: "encryptionkey@0"
I20260927 00:47:09.416214 3647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c8cffc76435d41958287ed0e34cab65f"
format_stamp: "Formatted at 2026-09-27 00:47:09 on dist-test-slave-86rs"
server_key: "ea67521ce77100ec086adc9880cb1963"
server_key_iv: "4b5972a026f7e6ab9043813ed53554fd"
server_key_version: "encryptionkey@0"
I20260927 00:47:09.421551 3647 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 00:47:09.424811 3662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:09.426600 3647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:47:09.426761 3647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "c8cffc76435d41958287ed0e34cab65f"
format_stamp: "Formatted at 2026-09-27 00:47:09 on dist-test-slave-86rs"
server_key: "ea67521ce77100ec086adc9880cb1963"
server_key_iv: "4b5972a026f7e6ab9043813ed53554fd"
server_key_version: "encryptionkey@0"
I20260927 00:47:09.426908 3647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:09.453662 3647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:09.458292 3647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:09.458580 3647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:09.459408 3647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:09.460628 3647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:09.460724 3647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:09.460786 3647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:09.460813 3647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:09.472502 3647 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33607
I20260927 00:47:09.472512 3775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33607 every 8 connection(s)
I20260927 00:47:09.473810 3647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:47:09.482549 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 3647
I20260927 00:47:09.482715 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:47:09.483052 392 external_mini_cluster.cc:1428] Setting key c04d7836cd5b2ac62240f6b2aae13349
Sep 27 00:47:09 dist-test-slave-86rs krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:09.491652 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:09.476506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40653 (local address 127.0.98.62:39651)
0927 00:47:09.476728 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:09.476733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:09.477887 (+ 1154us) server_negotiation.cc:407] Connection header received
0927 00:47:09.479547 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:09.479550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:09.479618 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:09.479702 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:09.481368 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:09.481997 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:09.483542 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:09.483845 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:09.483960 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:09.487441 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:09.487445 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:09.487447 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:09.487478 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:09.489160 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:09.489783 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:09.489786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:09.489788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:09.489892 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:09.490289 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:09.490293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:09.490295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:09.490526 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:09.490643 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:09.491271 (+ 628us) server_negotiation.cc:299] Negotiation successful
0927 00:47:09.491434 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 00:47:09.492971 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:09.477087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39651 (local address 127.0.98.2:40653)
0927 00:47:09.477694 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:09.477750 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:09.479286 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:09.479916 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:09.479924 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:09.480553 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:09.481193 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:09.481209 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:09.482179 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:09.482182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:09.483371 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:09.483381 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:09.484058 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:09.484062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:09.484331 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:09.485221 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:09.485242 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:09.487214 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:09.489308 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:09.489316 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:09.489347 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:09.489656 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:09.489997 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:09.490000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:09.490002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:09.490182 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:09.490633 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:09.490642 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:09.490980 (+ 338us) client_negotiation.cc:769] Sending connection context
0927 00:47:09.491219 (+ 239us) client_negotiation.cc:240] Negotiation successful
0927 00:47:09.491657 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":151,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:47:09.494516 3776 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39651
I20260927 00:47:09.494853 3776 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:09.495573 3776 heartbeater.cc:511] Master 127.0.98.62:39651 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:47:09.497143 3451 ts_manager.cc:194] Registered new tserver with Master: c8cffc76435d41958287ed0e34cab65f (127.0.98.2:33607)
I20260927 00:47:09.498003 3451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:47:09.515575 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39651
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:09.649910 3781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:09.650308 3781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:09.650358 3781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:09.655078 3781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:09.655184 3781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:09.655282 3781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:47:09.661722 3781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:09.663298 3781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:09.664812 3781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:47:09.677191 3781 server_base.cc:1034] running on GCE node
W20260927 00:47:09.677110 3786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:09.677119 3789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:09.677119 3787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:09.678000 3781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:09.678881 3781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:09.680066 3781 hybrid_clock.cc:648] HybridClock initialized: now 1790470029680017 us; error 64 us; skew 500 ppm
Sep 27 00:47:09 dist-test-slave-86rs krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:09.683722 3781 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:47:09.685051 3781 webserver.cc:460] Webserver started at http://127.0.98.3:44033/ using document root <none> and password file <none>
I20260927 00:47:09.685818 3781 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:09.685889 3781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:09.686158 3781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:09.688941 3781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/data/instance:
uuid: "534e7e5c03704ea1b570e4dc3fd8953a"
format_stamp: "Formatted at 2026-09-27 00:47:09 on dist-test-slave-86rs"
server_key: "e9a93e8c3a4085626975dfbec480fbd6"
server_key_iv: "321e476b882cc89a00abae35d08b954d"
server_key_version: "encryptionkey@0"
I20260927 00:47:09.689642 3781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/wal/instance:
uuid: "534e7e5c03704ea1b570e4dc3fd8953a"
format_stamp: "Formatted at 2026-09-27 00:47:09 on dist-test-slave-86rs"
server_key: "e9a93e8c3a4085626975dfbec480fbd6"
server_key_iv: "321e476b882cc89a00abae35d08b954d"
server_key_version: "encryptionkey@0"
I20260927 00:47:09.694562 3781 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 00:47:09.697815 3796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:09.699636 3781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:47:09.699798 3781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "534e7e5c03704ea1b570e4dc3fd8953a"
format_stamp: "Formatted at 2026-09-27 00:47:09 on dist-test-slave-86rs"
server_key: "e9a93e8c3a4085626975dfbec480fbd6"
server_key_iv: "321e476b882cc89a00abae35d08b954d"
server_key_version: "encryptionkey@0"
I20260927 00:47:09.699940 3781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:09.717126 3781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:09.721387 3781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:09.721642 3781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:09.722429 3781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:09.723502 3781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:09.723551 3781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:09.723642 3781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:09.723682 3781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:09.734965 3781 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43613
I20260927 00:47:09.734988 3909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43613 every 8 connection(s)
I20260927 00:47:09.736199 3781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:47:09.744410 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 3781
I20260927 00:47:09.744541 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790470006085796-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:47:09.744876 392 external_mini_cluster.cc:1428] Setting key c38314a6106aaf48435ff594eeaad1fc
Sep 27 00:47:09 dist-test-slave-86rs krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:09.754639 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:09.738874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43997 (local address 127.0.98.62:39651)
0927 00:47:09.739076 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:09.739080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:09.740268 (+ 1188us) server_negotiation.cc:407] Connection header received
0927 00:47:09.742065 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:09.742070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:09.742153 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:09.742285 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:09.743907 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:09.744714 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:09.746392 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:09.746663 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:09.746793 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:09.750094 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:09.750099 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:09.750102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:09.750136 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:09.751900 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:09.752700 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:09.752705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:09.752706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:09.752814 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:09.753316 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:09.753319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:09.753321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:09.753553 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:09.753641 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:09.754316 (+ 675us) server_negotiation.cc:299] Negotiation successful
0927 00:47:09.754473 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 00:47:09.755869 3912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:09.739350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39651 (local address 127.0.98.3:43997)
0927 00:47:09.740021 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:09.740084 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:09.741759 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:09.742545 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:09.742554 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:09.743069 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:09.743719 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:09.743737 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:09.744870 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:09.744877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:09.746251 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:09.746263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:09.746847 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:09.746852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:09.747091 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:09.747885 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:09.747908 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:09.749837 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:09.752104 (+ 2267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:09.752112 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:09.752140 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:09.752561 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:09.752920 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:09.752923 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:09.752925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:09.753208 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:09.753651 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:09.753659 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:09.754004 (+ 345us) client_negotiation.cc:769] Sending connection context
0927 00:47:09.754230 (+ 226us) client_negotiation.cc:240] Negotiation successful
0927 00:47:09.754605 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":152,"threads_started":1}
I20260927 00:47:09.757596 3910 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39651
I20260927 00:47:09.757995 3910 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:09.758884 3910 heartbeater.cc:511] Master 127.0.98.62:39651 requested a full tablet report, sending...
I20260927 00:47:09.760479 3451 ts_manager.cc:194] Registered new tserver with Master: 534e7e5c03704ea1b570e4dc3fd8953a (127.0.98.3:43613)
I20260927 00:47:09.761247 3451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43997
I20260927 00:47:09.772858 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 00:47:09 dist-test-slave-86rs krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470029, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 00:47:09 dist-test-slave-86rs krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470029, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:09.804659 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:09.793372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33448 (local address 127.0.98.62:39651)
0927 00:47:09.793551 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:09.793557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:09.793766 (+ 209us) server_negotiation.cc:407] Connection header received
0927 00:47:09.793970 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:09.793974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:09.794038 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:09.794181 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:09.795053 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:09.795679 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:09.796759 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:09.797010 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:09.797142 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:09.800520 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:09.800524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:09.800527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:09.800567 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:09.802386 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:09.802969 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:09.802973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:09.802975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:09.803082 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:09.803416 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:09.803421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:09.803422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:09.803766 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:09.803937 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:09.804288 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 00:47:09.804441 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 00:47:09.805181 3451 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33448
I20260927 00:47:09.806197 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 3513
I20260927 00:47:09.813432 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 3647
I20260927 00:47:09.820216 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 3781
I20260927 00:47:09.826913 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 3419
2026-09-27T00:47:09Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2425 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:47:09 dist-test-slave-86rs krb5kdc[3922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:47:09 dist-test-slave-86rs krb5kdc[3922](info): set up 2 sockets
Sep 27 00:47:09 dist-test-slave-86rs krb5kdc[3922](info): commencing operation
krb5kdc: starting...
W20260927 00:47:10.884872 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:47:10 dist-test-slave-86rs krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:47:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:47:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:47:11.032124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44211
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44211
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:11.176434 3935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:11.176873 3935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:11.176971 3935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:11.181483 3935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:47:11.181566 3935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:11.181587 3935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:11.181605 3935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:47:11.181622 3935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:47:11.187731 3935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:11.189406 3935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:11.190869 3935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:11.201298 3940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:11.201303 3941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:11.201298 3943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:11.201718 3935 server_base.cc:1034] running on GCE node
I20260927 00:47:11.202503 3935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:11.204002 3935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:11.205170 3935 hybrid_clock.cc:648] HybridClock initialized: now 1790470031205165 us; error 67 us; skew 500 ppm
Sep 27 00:47:11 dist-test-slave-86rs krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:11.213199 3935 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:47:11.214444 3935 webserver.cc:460] Webserver started at http://127.0.98.62:46403/ using document root <none> and password file <none>
I20260927 00:47:11.215224 3935 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:11.215327 3935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:11.215617 3935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:11.218350 3935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "2ada6835a5c24c8fa1e8839c4fd638ff"
format_stamp: "Formatted at 2026-09-27 00:47:11 on dist-test-slave-86rs"
server_key: "643c5bf809ad83e8a3fab202ee340211"
server_key_iv: "1a09e9feef6b6892b4e552f45ab47865"
server_key_version: "encryptionkey@0"
I20260927 00:47:11.219055 3935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "2ada6835a5c24c8fa1e8839c4fd638ff"
format_stamp: "Formatted at 2026-09-27 00:47:11 on dist-test-slave-86rs"
server_key: "643c5bf809ad83e8a3fab202ee340211"
server_key_iv: "1a09e9feef6b6892b4e552f45ab47865"
server_key_version: "encryptionkey@0"
I20260927 00:47:11.223831 3935 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 00:47:11.227075 3950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:11.228969 3935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:47:11.229154 3935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "2ada6835a5c24c8fa1e8839c4fd638ff"
format_stamp: "Formatted at 2026-09-27 00:47:11 on dist-test-slave-86rs"
server_key: "643c5bf809ad83e8a3fab202ee340211"
server_key_iv: "1a09e9feef6b6892b4e552f45ab47865"
server_key_version: "encryptionkey@0"
I20260927 00:47:11.229313 3935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:11.245738 3935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:11.250069 3935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:11.250320 3935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:11.259531 3935 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44211
I20260927 00:47:11.259585 4002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44211 every 8 connection(s)
I20260927 00:47:11.260893 3935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:47:11.264976 4003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:11.267668 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 3935
I20260927 00:47:11.267831 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:47:11.268206 392 external_mini_cluster.cc:1428] Setting key 4e1671d22387a9c289d09828c41e283b
I20260927 00:47:11.273995 4003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ada6835a5c24c8fa1e8839c4fd638ff: Bootstrap starting.
I20260927 00:47:11.277556 4003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ada6835a5c24c8fa1e8839c4fd638ff: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:11.278649 4003 log.cc:824] T 00000000000000000000000000000000 P 2ada6835a5c24c8fa1e8839c4fd638ff: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:47:11 dist-test-slave-86rs krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:11.281924 4003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ada6835a5c24c8fa1e8839c4fd638ff: No bootstrap required, opened a new log
I20260927 00:47:11.285877 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:11.270199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48210 (local address 127.0.98.62:44211)
0927 00:47:11.270815 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:11.270835 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:11.270884 (+ 49us) server_negotiation.cc:407] Connection header received
0927 00:47:11.271718 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:11.271743 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:11.272298 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:11.272782 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:11.273782 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:11.274957 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:11.275948 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:11.276624 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:11.276843 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:11.280155 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:11.280161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:11.280181 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:11.280214 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:11.282372 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:11.282888 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:11.282896 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:11.282910 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:11.283043 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:11.283366 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:11.283371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:11.283373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:11.283884 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:11.284102 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:11.284486 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 00:47:11.284860 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":198,"threads_started":1}
I20260927 00:47:11.286840 4003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ada6835a5c24c8fa1e8839c4fd638ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ada6835a5c24c8fa1e8839c4fd638ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44211 } }
I20260927 00:47:11.287154 4003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ada6835a5c24c8fa1e8839c4fd638ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:11.287271 4003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ada6835a5c24c8fa1e8839c4fd638ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ada6835a5c24c8fa1e8839c4fd638ff, State: Initialized, Role: FOLLOWER
I20260927 00:47:11.287909 4003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ada6835a5c24c8fa1e8839c4fd638ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ada6835a5c24c8fa1e8839c4fd638ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44211 } }
I20260927 00:47:11.288111 4003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ada6835a5c24c8fa1e8839c4fd638ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:47:11.288187 4003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ada6835a5c24c8fa1e8839c4fd638ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:47:11.288329 4003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ada6835a5c24c8fa1e8839c4fd638ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:11.289754 4003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ada6835a5c24c8fa1e8839c4fd638ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ada6835a5c24c8fa1e8839c4fd638ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44211 } }
I20260927 00:47:11.290153 4003 leader_election.cc:302] T 00000000000000000000000000000000 P 2ada6835a5c24c8fa1e8839c4fd638ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ada6835a5c24c8fa1e8839c4fd638ff; no voters:
I20260927 00:47:11.290602 4003 leader_election.cc:288] T 00000000000000000000000000000000 P 2ada6835a5c24c8fa1e8839c4fd638ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:47:11.290738 4008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ada6835a5c24c8fa1e8839c4fd638ff [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:11.290977 4008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ada6835a5c24c8fa1e8839c4fd638ff [term 1 LEADER]: Becoming Leader. State: Replica: 2ada6835a5c24c8fa1e8839c4fd638ff, State: Running, Role: LEADER
I20260927 00:47:11.291335 4008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ada6835a5c24c8fa1e8839c4fd638ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ada6835a5c24c8fa1e8839c4fd638ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44211 } }
I20260927 00:47:11.292418 4003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ada6835a5c24c8fa1e8839c4fd638ff [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:47:11.293450 4010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ada6835a5c24c8fa1e8839c4fd638ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ada6835a5c24c8fa1e8839c4fd638ff. Latest consensus state: current_term: 1 leader_uuid: "2ada6835a5c24c8fa1e8839c4fd638ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ada6835a5c24c8fa1e8839c4fd638ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44211 } } }
I20260927 00:47:11.293692 4010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ada6835a5c24c8fa1e8839c4fd638ff [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:11.294207 4017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:47:11.293480 4009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ada6835a5c24c8fa1e8839c4fd638ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ada6835a5c24c8fa1e8839c4fd638ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ada6835a5c24c8fa1e8839c4fd638ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44211 } } }
I20260927 00:47:11.294363 4009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ada6835a5c24c8fa1e8839c4fd638ff [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:11.298200 4017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:47:11.306107 4017 catalog_manager.cc:1347] Generated new cluster ID: 352abace2a9944f684e9ede54bdafce6
I20260927 00:47:11.306207 4017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:47:11.321594 4017 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:47:11.322976 4017 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:47:11.332957 4017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ada6835a5c24c8fa1e8839c4fd638ff: Generated new TSK 0
I20260927 00:47:11.334360 4017 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:47:11.379559 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44211
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:11.513994 4029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:11.514426 4029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:11.514504 4029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:11.518976 4029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:11.519109 4029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:11.519222 4029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:47:11.525259 4029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:11.526829 4029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:11.528304 4029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:11.540653 4037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:11.540653 4034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:11.540653 4035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:11.541162 4029 server_base.cc:1034] running on GCE node
I20260927 00:47:11.541671 4029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:11.542528 4029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:11.543735 4029 hybrid_clock.cc:648] HybridClock initialized: now 1790470031543714 us; error 43 us; skew 500 ppm
Sep 27 00:47:11 dist-test-slave-86rs krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:11.547044 4029 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:47:11.548179 4029 webserver.cc:460] Webserver started at http://127.0.98.1:41283/ using document root <none> and password file <none>
I20260927 00:47:11.548961 4029 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:11.549031 4029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:11.549292 4029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:11.551915 4029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-0/data/instance:
uuid: "7d8217ba4a5c4e018a43ea5c64f145f2"
format_stamp: "Formatted at 2026-09-27 00:47:11 on dist-test-slave-86rs"
server_key: "e3fd39931c788e533b74d8ea0475fadd"
server_key_iv: "15f145f053cb8aa9b26934d1c40b8d01"
server_key_version: "encryptionkey@0"
I20260927 00:47:11.552685 4029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7d8217ba4a5c4e018a43ea5c64f145f2"
format_stamp: "Formatted at 2026-09-27 00:47:11 on dist-test-slave-86rs"
server_key: "e3fd39931c788e533b74d8ea0475fadd"
server_key_iv: "15f145f053cb8aa9b26934d1c40b8d01"
server_key_version: "encryptionkey@0"
I20260927 00:47:11.557703 4029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 00:47:11.561223 4044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:11.563136 4029 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:47:11.563294 4029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "7d8217ba4a5c4e018a43ea5c64f145f2"
format_stamp: "Formatted at 2026-09-27 00:47:11 on dist-test-slave-86rs"
server_key: "e3fd39931c788e533b74d8ea0475fadd"
server_key_iv: "15f145f053cb8aa9b26934d1c40b8d01"
server_key_version: "encryptionkey@0"
I20260927 00:47:11.563503 4029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:11.599748 4029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:11.604697 4029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:11.605011 4029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:11.605885 4029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:11.607090 4029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:11.607149 4029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:11.607225 4029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:11.607262 4029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:11.619069 4029 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37495
I20260927 00:47:11.619087 4157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37495 every 8 connection(s)
I20260927 00:47:11.620347 4029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:47:11.628922 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 4029
I20260927 00:47:11.629127 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:47:11.629482 392 external_mini_cluster.cc:1428] Setting key c9d713b93652a479115ef2c02e5fd0f7
Sep 27 00:47:11 dist-test-slave-86rs krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:11.639281 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:11.622853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33299 (local address 127.0.98.62:44211)
0927 00:47:11.623030 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:11.623035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:11.624178 (+ 1143us) server_negotiation.cc:407] Connection header received
0927 00:47:11.625896 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:11.625900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:11.625975 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:11.626095 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:11.627644 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:11.628318 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:11.629761 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:11.630025 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:11.630147 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:11.633882 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:11.633890 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:11.633894 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:11.633937 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:11.636167 (+ 2230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:11.636995 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:11.637002 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:11.637006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:11.637130 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:11.637661 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:11.637666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:11.637668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:11.638007 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:11.638141 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:11.638847 (+ 706us) server_negotiation.cc:299] Negotiation successful
0927 00:47:11.639042 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 00:47:11.640506 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:11.623364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44211 (local address 127.0.98.1:33299)
0927 00:47:11.623943 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:11.624001 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:11.625620 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:11.626325 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:11.626339 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:11.626808 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:11.627473 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:11.627489 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:11.628456 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:11.628465 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:11.629638 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:11.629648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:11.630206 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:11.630210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:11.630458 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:11.631303 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:11.631343 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:11.633600 (+ 2257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:11.636370 (+ 2770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:11.636377 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:11.636399 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:11.636847 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:11.637279 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:11.637283 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:11.637284 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:11.637511 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:11.638138 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:11.638145 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:11.638509 (+ 364us) client_negotiation.cc:769] Sending connection context
0927 00:47:11.638757 (+ 248us) client_negotiation.cc:240] Negotiation successful
0927 00:47:11.639148 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":153,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:47:11.642398 4158 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44211
I20260927 00:47:11.642828 4158 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:11.643652 4158 heartbeater.cc:511] Master 127.0.98.62:44211 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:47:11.645978 3966 ts_manager.cc:194] Registered new tserver with Master: 7d8217ba4a5c4e018a43ea5c64f145f2 (127.0.98.1:37495)
I20260927 00:47:11.647439 3966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33299
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:47:11.663846 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44211
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:11.800437 4163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:11.800796 4163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:11.800859 4163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:11.805454 4163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:11.805565 4163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:11.805671 4163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:47:11.811533 4163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:11.813133 4163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:11.814531 4163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:47:11.826452 4163 server_base.cc:1034] running on GCE node
W20260927 00:47:11.826375 4169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:11.826393 4168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:11.826375 4171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:11.827278 4163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:11.828120 4163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:11.829313 4163 hybrid_clock.cc:648] HybridClock initialized: now 1790470031829288 us; error 44 us; skew 500 ppm
Sep 27 00:47:11 dist-test-slave-86rs krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:11.832762 4163 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:47:11.834043 4163 webserver.cc:460] Webserver started at http://127.0.98.2:34677/ using document root <none> and password file <none>
I20260927 00:47:11.834828 4163 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:11.834910 4163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:11.835141 4163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:11.837906 4163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-1/data/instance:
uuid: "4cb4578308f647d7ba0e33620be5d74c"
format_stamp: "Formatted at 2026-09-27 00:47:11 on dist-test-slave-86rs"
server_key: "8336b0d7d87391dda34acbd2e99a6ea4"
server_key_iv: "a00af239289ff543c213b9717c38dace"
server_key_version: "encryptionkey@0"
I20260927 00:47:11.838589 4163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4cb4578308f647d7ba0e33620be5d74c"
format_stamp: "Formatted at 2026-09-27 00:47:11 on dist-test-slave-86rs"
server_key: "8336b0d7d87391dda34acbd2e99a6ea4"
server_key_iv: "a00af239289ff543c213b9717c38dace"
server_key_version: "encryptionkey@0"
I20260927 00:47:11.843696 4163 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 00:47:11.846969 4178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:11.848682 4163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 00:47:11.848814 4163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "4cb4578308f647d7ba0e33620be5d74c"
format_stamp: "Formatted at 2026-09-27 00:47:11 on dist-test-slave-86rs"
server_key: "8336b0d7d87391dda34acbd2e99a6ea4"
server_key_iv: "a00af239289ff543c213b9717c38dace"
server_key_version: "encryptionkey@0"
I20260927 00:47:11.848927 4163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:11.864936 4163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:11.869196 4163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:11.869395 4163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:11.870172 4163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:11.871181 4163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:11.871232 4163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:11.871280 4163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:11.871296 4163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:11.882052 4163 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38527
I20260927 00:47:11.882098 4291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38527 every 8 connection(s)
I20260927 00:47:11.883309 4163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:47:11.892608 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 4163
I20260927 00:47:11.892791 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:47:11.893199 392 external_mini_cluster.cc:1428] Setting key a91c9afdf259bbf78960e1f8c3b0448e
Sep 27 00:47:11 dist-test-slave-86rs krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:11.902022 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:11.885822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56127 (local address 127.0.98.62:44211)
0927 00:47:11.885999 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:11.886003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:11.887098 (+ 1095us) server_negotiation.cc:407] Connection header received
0927 00:47:11.888860 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:11.888864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:11.888928 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:11.889013 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:11.890540 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:11.891136 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:11.892654 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:11.893126 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:11.893471 (+ 345us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:11.897308 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:11.897313 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:11.897316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:11.897355 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:11.899107 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:11.899846 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:11.899851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:11.899853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:11.899965 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:11.900509 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:11.900512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:11.900514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:11.900782 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:11.900903 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:11.901632 (+ 729us) server_negotiation.cc:299] Negotiation successful
0927 00:47:11.901809 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 00:47:11.903280 4294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:11.886292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44211 (local address 127.0.98.2:56127)
0927 00:47:11.886888 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:11.886943 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:11.888597 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:11.889241 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:11.889251 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:11.889750 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:11.890382 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:11.890398 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:11.891271 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:11.891275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:11.892452 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:11.892463 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:11.893515 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:11.893524 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:11.893905 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:11.894803 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:11.894832 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:11.897057 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:11.899284 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:11.899292 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:11.899318 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:11.899712 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:11.900077 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:11.900080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:11.900082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:11.900381 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:11.900905 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:11.900917 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:11.901307 (+ 390us) client_negotiation.cc:769] Sending connection context
0927 00:47:11.901570 (+ 263us) client_negotiation.cc:240] Negotiation successful
0927 00:47:11.902051 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":160,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:47:11.905079 4292 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44211
I20260927 00:47:11.905439 4292 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:11.906175 4292 heartbeater.cc:511] Master 127.0.98.62:44211 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:47:11.907792 3966 ts_manager.cc:194] Registered new tserver with Master: 4cb4578308f647d7ba0e33620be5d74c (127.0.98.2:38527)
I20260927 00:47:11.908824 3966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:47:11.925429 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44211
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:12.064498 4297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:12.064883 4297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:12.064968 4297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:12.069442 4297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:12.069540 4297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:12.069635 4297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:47:12.075543 4297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:12.077078 4297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:12.078456 4297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:12.090287 4305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:12.090265 4302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:12.090456 4297 server_base.cc:1034] running on GCE node
W20260927 00:47:12.090689 4303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:12.091248 4297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:12.092257 4297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:12.093443 4297 hybrid_clock.cc:648] HybridClock initialized: now 1790470032093392 us; error 66 us; skew 500 ppm
Sep 27 00:47:12 dist-test-slave-86rs krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:12.097086 4297 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:47:12.098311 4297 webserver.cc:460] Webserver started at http://127.0.98.3:40689/ using document root <none> and password file <none>
I20260927 00:47:12.099085 4297 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:12.099160 4297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:12.099438 4297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:12.102401 4297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-2/data/instance:
uuid: "d7d65d5f30ac421c87ca4b4b681d6751"
format_stamp: "Formatted at 2026-09-27 00:47:12 on dist-test-slave-86rs"
server_key: "0161c1773a88a7f11a2edb65cd006a0b"
server_key_iv: "6054907f0b2ec98afbc65ad2242d627d"
server_key_version: "encryptionkey@0"
I20260927 00:47:12.103222 4297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d7d65d5f30ac421c87ca4b4b681d6751"
format_stamp: "Formatted at 2026-09-27 00:47:12 on dist-test-slave-86rs"
server_key: "0161c1773a88a7f11a2edb65cd006a0b"
server_key_iv: "6054907f0b2ec98afbc65ad2242d627d"
server_key_version: "encryptionkey@0"
I20260927 00:47:12.108700 4297 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 00:47:12.112349 4312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:12.114640 4297 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:47:12.114884 4297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "d7d65d5f30ac421c87ca4b4b681d6751"
format_stamp: "Formatted at 2026-09-27 00:47:12 on dist-test-slave-86rs"
server_key: "0161c1773a88a7f11a2edb65cd006a0b"
server_key_iv: "6054907f0b2ec98afbc65ad2242d627d"
server_key_version: "encryptionkey@0"
I20260927 00:47:12.115069 4297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:12.147929 4297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:12.153291 4297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:12.153585 4297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:12.154485 4297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:12.155743 4297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:12.155810 4297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:12.155864 4297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:12.155880 4297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:12.168545 4297 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37163
I20260927 00:47:12.168565 4425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37163 every 8 connection(s)
I20260927 00:47:12.169983 4297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:47:12.175064 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 4297
I20260927 00:47:12.175268 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790470006085796-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:47:12.175698 392 external_mini_cluster.cc:1428] Setting key 2b4beb5d10a28ddb3004f14fe72a4021
Sep 27 00:47:12 dist-test-slave-86rs krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:12.190644 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:12.172644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52211 (local address 127.0.98.62:44211)
0927 00:47:12.172856 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:12.172861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:12.174254 (+ 1393us) server_negotiation.cc:407] Connection header received
0927 00:47:12.176719 (+ 2465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:12.176723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:12.176799 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:12.176886 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:12.179269 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:12.180038 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:12.181736 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:12.182065 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:12.182191 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:12.185510 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:12.185515 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:12.185518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:12.185560 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:12.187525 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:12.188183 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:12.188187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:12.188189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:12.188372 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:12.188881 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:12.188889 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:12.188893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:12.189215 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:12.189348 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:12.190151 (+ 803us) server_negotiation.cc:299] Negotiation successful
0927 00:47:12.190341 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 00:47:12.191870 4428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:12.173258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44211 (local address 127.0.98.3:52211)
0927 00:47:12.174032 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:12.174090 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:12.176390 (+ 2300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:12.177230 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:12.177246 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:12.178109 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:12.179084 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:12.179104 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:12.180199 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:12.180202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:12.181556 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:12.181569 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:12.182288 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:12.182293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:12.182518 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:12.183344 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:12.183366 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:12.185284 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:12.187681 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:12.187689 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:12.187715 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:12.188042 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:12.188464 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:12.188467 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:12.188469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:12.188721 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:12.189343 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:12.189352 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:12.189798 (+ 446us) client_negotiation.cc:769] Sending connection context
0927 00:47:12.190060 (+ 262us) client_negotiation.cc:240] Negotiation successful
0927 00:47:12.190564 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":189,"threads_started":1}
I20260927 00:47:12.193369 4426 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44211
I20260927 00:47:12.193756 4426 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:12.194424 4426 heartbeater.cc:511] Master 127.0.98.62:44211 requested a full tablet report, sending...
I20260927 00:47:12.195801 3966 ts_manager.cc:194] Registered new tserver with Master: d7d65d5f30ac421c87ca4b4b681d6751 (127.0.98.3:37163)
I20260927 00:47:12.196532 3966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52211
I20260927 00:47:12.203677 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 00:47:12 dist-test-slave-86rs krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:47:12 dist-test-slave-86rs krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:47:12.296527 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:12.284985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43712 (local address 127.0.98.1:37495)
0927 00:47:12.285579 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:12.285587 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:12.285615 (+ 28us) server_negotiation.cc:407] Connection header received
0927 00:47:12.285678 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:12.285685 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:12.285994 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:12.286189 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:12.287093 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:12.288025 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:12.289084 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:12.289551 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:12.289712 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:12.292695 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:12.292707 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:12.292726 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:12.292776 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:12.294610 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:12.295070 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:12.295077 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:12.295078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:12.295186 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:12.295470 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:12.295472 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:12.295474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:12.295757 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:12.295942 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:12.296136 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 00:47:12.296356 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":87,"threads_started":1}
Sep 27 00:47:12 dist-test-slave-86rs krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:12.310915 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:12.301200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48216 (local address 127.0.98.62:44211)
0927 00:47:12.301391 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:12.301396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:12.301527 (+ 131us) server_negotiation.cc:407] Connection header received
0927 00:47:12.301710 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:12.301713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:12.301763 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:12.301855 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:12.302607 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:12.303186 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:12.304077 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:12.304366 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:12.304481 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:12.307363 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:12.307367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:12.307370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:12.307416 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:12.309020 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:12.309479 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:12.309483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:12.309485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:12.309571 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:12.309871 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:12.309875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:12.309877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:12.310155 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:12.310310 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:12.310569 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 00:47:12.310724 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260927 00:47:12.311522 3966 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48216
I20260927 00:47:12.312354 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 4029
I20260927 00:47:12.319155 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 4163
I20260927 00:47:12.326218 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 4297
I20260927 00:47:12.333886 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 3935
2026-09-27T00:47:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2507 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:47:12 dist-test-slave-86rs krb5kdc[4438](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:47:12 dist-test-slave-86rs krb5kdc[4438](info): set up 2 sockets
Sep 27 00:47:12 dist-test-slave-86rs krb5kdc[4438](info): commencing operation
krb5kdc: starting...
W20260927 00:47:13.473214 392 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:47:13 dist-test-slave-86rs krb5kdc[4438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:47:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:47:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:47:13.632423 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35441
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:13.769863 4451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:13.770257 4451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:13.770308 4451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:13.774641 4451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:47:13.774714 4451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:13.774734 4451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:13.774751 4451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:47:13.774766 4451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:47:13.780635 4451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:13.782222 4451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:13.783666 4451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:13.794890 4456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:13.794886 4459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:13.794895 4457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:13.795419 4451 server_base.cc:1034] running on GCE node
I20260927 00:47:13.796211 4451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:13.797958 4451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:13.799206 4451 hybrid_clock.cc:648] HybridClock initialized: now 1790470033799186 us; error 44 us; skew 500 ppm
Sep 27 00:47:13 dist-test-slave-86rs krb5kdc[4438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:13.802983 4451 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:47:13.804327 4451 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:13.804452 4451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:13.804831 4451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:13.807611 4451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "352bee5d05c34231ba78730b62243a77"
format_stamp: "Formatted at 2026-09-27 00:47:13 on dist-test-slave-86rs"
server_key: "1fb3fa0a1207698fef77bb2a8c3b8ad5"
server_key_iv: "3c2a48b9cf584befe8424f874bfd16bb"
server_key_version: "encryptionkey@0"
I20260927 00:47:13.808473 4451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "352bee5d05c34231ba78730b62243a77"
format_stamp: "Formatted at 2026-09-27 00:47:13 on dist-test-slave-86rs"
server_key: "1fb3fa0a1207698fef77bb2a8c3b8ad5"
server_key_iv: "3c2a48b9cf584befe8424f874bfd16bb"
server_key_version: "encryptionkey@0"
I20260927 00:47:13.813939 4451 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 00:47:13.817406 4465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:13.819355 4451 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:47:13.819573 4451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "352bee5d05c34231ba78730b62243a77"
format_stamp: "Formatted at 2026-09-27 00:47:13 on dist-test-slave-86rs"
server_key: "1fb3fa0a1207698fef77bb2a8c3b8ad5"
server_key_iv: "3c2a48b9cf584befe8424f874bfd16bb"
server_key_version: "encryptionkey@0"
I20260927 00:47:13.819742 4451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:13.836453 4451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:13.841063 4451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:13.841334 4451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:13.850090 4451 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35441
I20260927 00:47:13.850405 4517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35441 every 8 connection(s)
I20260927 00:47:13.850965 4451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:47:13.852465 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 4451
I20260927 00:47:13.852609 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:47:13.852957 392 external_mini_cluster.cc:1428] Setting key 3599d020382d43a5c55d9100a611a0ff
I20260927 00:47:13.855330 4518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:13.862736 4518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 352bee5d05c34231ba78730b62243a77: Bootstrap starting.
Sep 27 00:47:13 dist-test-slave-86rs krb5kdc[4438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:13.866572 4518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 352bee5d05c34231ba78730b62243a77: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:13.867717 4518 log.cc:824] T 00000000000000000000000000000000 P 352bee5d05c34231ba78730b62243a77: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:13.870631 4518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 352bee5d05c34231ba78730b62243a77: No bootstrap required, opened a new log
I20260927 00:47:13.870664 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:13.854705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35136 (local address 127.0.98.62:35441)
0927 00:47:13.855273 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:13.855293 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:13.855341 (+ 48us) server_negotiation.cc:407] Connection header received
0927 00:47:13.856194 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:13.856216 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:13.856763 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:13.857199 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:13.858163 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:13.859357 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:13.860366 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:13.860933 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:13.861166 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:13.864517 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:13.864527 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:13.864552 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:13.864603 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:13.866851 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:13.867374 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:13.867381 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:13.867388 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:13.867566 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:13.867961 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:13.867975 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:13.867976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:13.868564 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:13.868731 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:13.869125 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 00:47:13.869492 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":195,"threads_started":1}
I20260927 00:47:13.875149 4518 raft_consensus.cc:348] T 00000000000000000000000000000000 P 352bee5d05c34231ba78730b62243a77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352bee5d05c34231ba78730b62243a77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35441 } }
I20260927 00:47:13.875428 4518 raft_consensus.cc:374] T 00000000000000000000000000000000 P 352bee5d05c34231ba78730b62243a77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:13.875483 4518 raft_consensus.cc:729] T 00000000000000000000000000000000 P 352bee5d05c34231ba78730b62243a77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 352bee5d05c34231ba78730b62243a77, State: Initialized, Role: FOLLOWER
I20260927 00:47:13.875994 4518 consensus_queue.cc:260] T 00000000000000000000000000000000 P 352bee5d05c34231ba78730b62243a77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352bee5d05c34231ba78730b62243a77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35441 } }
I20260927 00:47:13.876130 4518 raft_consensus.cc:388] T 00000000000000000000000000000000 P 352bee5d05c34231ba78730b62243a77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:47:13.876173 4518 raft_consensus.cc:482] T 00000000000000000000000000000000 P 352bee5d05c34231ba78730b62243a77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:47:13.876307 4518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 352bee5d05c34231ba78730b62243a77 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:13.877381 4518 raft_consensus.cc:504] T 00000000000000000000000000000000 P 352bee5d05c34231ba78730b62243a77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352bee5d05c34231ba78730b62243a77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35441 } }
I20260927 00:47:13.877750 4518 leader_election.cc:302] T 00000000000000000000000000000000 P 352bee5d05c34231ba78730b62243a77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 352bee5d05c34231ba78730b62243a77; no voters:
I20260927 00:47:13.878134 4518 leader_election.cc:288] T 00000000000000000000000000000000 P 352bee5d05c34231ba78730b62243a77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:47:13.878297 4523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 352bee5d05c34231ba78730b62243a77 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:13.878569 4523 raft_consensus.cc:686] T 00000000000000000000000000000000 P 352bee5d05c34231ba78730b62243a77 [term 1 LEADER]: Becoming Leader. State: Replica: 352bee5d05c34231ba78730b62243a77, State: Running, Role: LEADER
I20260927 00:47:13.878960 4523 consensus_queue.cc:237] T 00000000000000000000000000000000 P 352bee5d05c34231ba78730b62243a77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352bee5d05c34231ba78730b62243a77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35441 } }
I20260927 00:47:13.879784 4518 sys_catalog.cc:564] T 00000000000000000000000000000000 P 352bee5d05c34231ba78730b62243a77 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:47:13.881085 4524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 352bee5d05c34231ba78730b62243a77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "352bee5d05c34231ba78730b62243a77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352bee5d05c34231ba78730b62243a77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35441 } } }
I20260927 00:47:13.881227 4524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 352bee5d05c34231ba78730b62243a77 [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:13.881526 4525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 352bee5d05c34231ba78730b62243a77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 352bee5d05c34231ba78730b62243a77. Latest consensus state: current_term: 1 leader_uuid: "352bee5d05c34231ba78730b62243a77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352bee5d05c34231ba78730b62243a77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35441 } } }
I20260927 00:47:13.881614 4525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 352bee5d05c34231ba78730b62243a77 [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:13.881768 4532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:47:13.885617 4532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:47:13.893698 4532 catalog_manager.cc:1347] Generated new cluster ID: 84596b2ebcd749f8a859430d2175e0ed
I20260927 00:47:13.893836 4532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:47:13.911433 4532 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:47:13.913225 4532 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:47:13.921286 4532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 352bee5d05c34231ba78730b62243a77: Generated new TSK 0
I20260927 00:47:13.922544 4532 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:47:13.959856 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35441
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:14.104625 4544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:14.105072 4544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:14.105129 4544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:14.111258 4544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:14.111387 4544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:14.111466 4544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:47:14.118429 4544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:14.120078 4544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:14.121668 4544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:14.134603 4549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:14.134696 4550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:14.134696 4552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:14.135185 4544 server_base.cc:1034] running on GCE node
I20260927 00:47:14.135692 4544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:14.136654 4544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:14.137862 4544 hybrid_clock.cc:648] HybridClock initialized: now 1790470034137816 us; error 69 us; skew 500 ppm
Sep 27 00:47:14 dist-test-slave-86rs krb5kdc[4438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:14.141314 4544 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:47:14.142299 4544 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:14.142409 4544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:14.142710 4544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:14.145848 4544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-0/data/instance:
uuid: "09b07b7a3434483b9ca37871b266432a"
format_stamp: "Formatted at 2026-09-27 00:47:14 on dist-test-slave-86rs"
server_key: "5539203db7fa15021b1e9c7ab001a515"
server_key_iv: "32ad89bdfd5afe90cdc966d91925e9bf"
server_key_version: "encryptionkey@0"
I20260927 00:47:14.146651 4544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-0/wal/instance:
uuid: "09b07b7a3434483b9ca37871b266432a"
format_stamp: "Formatted at 2026-09-27 00:47:14 on dist-test-slave-86rs"
server_key: "5539203db7fa15021b1e9c7ab001a515"
server_key_iv: "32ad89bdfd5afe90cdc966d91925e9bf"
server_key_version: "encryptionkey@0"
I20260927 00:47:14.151830 4544 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 00:47:14.156196 4558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:14.158533 4544 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:47:14.158802 4544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "09b07b7a3434483b9ca37871b266432a"
format_stamp: "Formatted at 2026-09-27 00:47:14 on dist-test-slave-86rs"
server_key: "5539203db7fa15021b1e9c7ab001a515"
server_key_iv: "32ad89bdfd5afe90cdc966d91925e9bf"
server_key_version: "encryptionkey@0"
I20260927 00:47:14.159001 4544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:14.178169 4544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:14.183892 4544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:14.184264 4544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:14.185206 4544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:14.186477 4544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:14.186576 4544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:14.186679 4544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:14.186740 4544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:14.198352 4544 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33997
I20260927 00:47:14.198464 4671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33997 every 8 connection(s)
I20260927 00:47:14.199316 4544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:47:14.199795 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 4544
I20260927 00:47:14.199941 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:47:14.200410 392 external_mini_cluster.cc:1428] Setting key 7f130a179dd03f283134b6509a2b8f3f
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 27 00:47:14 dist-test-slave-86rs krb5kdc[4438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:14.222956 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:14.201947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45573 (local address 127.0.98.62:35441)
0927 00:47:14.202130 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:14.202134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:14.203699 (+ 1565us) server_negotiation.cc:407] Connection header received
0927 00:47:14.205859 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:14.205865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:14.205950 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:14.206071 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:14.208456 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.209095 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.211359 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.211635 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.211737 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:14.217921 (+ 6184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:14.217926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:14.217928 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:14.217967 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:14.219991 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:14.220841 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:14.220845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:14.220847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:14.220959 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:14.221459 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:14.221462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:14.221463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:14.221711 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:14.221838 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:14.222562 (+ 724us) server_negotiation.cc:299] Negotiation successful
0927 00:47:14.222752 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 00:47:14.224371 4674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:14.202706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35441 (local address 127.0.98.1:45573)
0927 00:47:14.203436 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:14.203534 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:14.205507 (+ 1973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:14.206466 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:14.206482 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:14.207194 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:14.208191 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:14.208256 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.209292 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.209298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:14.211184 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:14.211199 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.211856 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.211860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:14.212168 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:14.213066 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:14.213102 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:14.217564 (+ 4462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:14.220179 (+ 2615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:14.220187 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:14.220210 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:14.220676 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:14.221121 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:14.221123 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:14.221125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:14.221323 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:14.221836 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:14.221847 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:14.222244 (+ 397us) client_negotiation.cc:769] Sending connection context
0927 00:47:14.222495 (+ 251us) client_negotiation.cc:240] Negotiation successful
0927 00:47:14.222969 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":154,"threads_started":1}
I20260927 00:47:14.226397 4672 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35441
I20260927 00:47:14.226758 4672 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:14.227507 4672 heartbeater.cc:511] Master 127.0.98.62:35441 requested a full tablet report, sending...
I20260927 00:47:14.229894 4482 ts_manager.cc:194] Registered new tserver with Master: 09b07b7a3434483b9ca37871b266432a (127.0.98.1:33997)
I20260927 00:47:14.231503 4482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:47:14.236065 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35441
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:14.382207 4677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:14.382591 4677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:14.382647 4677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:14.387331 4677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:14.387437 4677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:14.387547 4677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:47:14.393983 4677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:14.395526 4677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:14.397270 4677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:14.410211 4683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:14.410187 4682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:14.410187 4685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:14.410312 4677 server_base.cc:1034] running on GCE node
I20260927 00:47:14.411235 4677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:14.412103 4677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:14.413287 4677 hybrid_clock.cc:648] HybridClock initialized: now 1790470034413262 us; error 42 us; skew 500 ppm
Sep 27 00:47:14 dist-test-slave-86rs krb5kdc[4438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:14.416862 4677 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:47:14.417848 4677 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:14.417914 4677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:14.418166 4677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:14.420778 4677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-1/data/instance:
uuid: "eb324ec8f05f41959fb97a96cbeb11a7"
format_stamp: "Formatted at 2026-09-27 00:47:14 on dist-test-slave-86rs"
server_key: "0830ffd7587f8572cb987195f7a27a6a"
server_key_iv: "a680aa6d3cc75d6cd1275befeae7c81c"
server_key_version: "encryptionkey@0"
I20260927 00:47:14.421515 4677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-1/wal/instance:
uuid: "eb324ec8f05f41959fb97a96cbeb11a7"
format_stamp: "Formatted at 2026-09-27 00:47:14 on dist-test-slave-86rs"
server_key: "0830ffd7587f8572cb987195f7a27a6a"
server_key_iv: "a680aa6d3cc75d6cd1275befeae7c81c"
server_key_version: "encryptionkey@0"
I20260927 00:47:14.426290 4677 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:47:14.429539 4691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:14.431282 4677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:47:14.431440 4677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "eb324ec8f05f41959fb97a96cbeb11a7"
format_stamp: "Formatted at 2026-09-27 00:47:14 on dist-test-slave-86rs"
server_key: "0830ffd7587f8572cb987195f7a27a6a"
server_key_iv: "a680aa6d3cc75d6cd1275befeae7c81c"
server_key_version: "encryptionkey@0"
I20260927 00:47:14.431591 4677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:14.450380 4677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:14.454231 4677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:14.454468 4677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:14.455292 4677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:14.456403 4677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:14.456452 4677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:14.456528 4677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:14.456562 4677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:14.467317 4677 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38449
I20260927 00:47:14.467335 4804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38449 every 8 connection(s)
I20260927 00:47:14.468083 4677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:47:14.475551 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 4677
I20260927 00:47:14.475703 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:47:14.476023 392 external_mini_cluster.cc:1428] Setting key 221ad5fd7255af58e1b25bbfdd885040
Sep 27 00:47:14 dist-test-slave-86rs krb5kdc[4438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:14.485375 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:14.470303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42559 (local address 127.0.98.62:35441)
0927 00:47:14.470498 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:14.470503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:14.471528 (+ 1025us) server_negotiation.cc:407] Connection header received
0927 00:47:14.473078 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:14.473081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:14.473142 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:14.473226 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:14.474645 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.475306 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.476907 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.477178 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.477303 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:14.480618 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:14.480622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:14.480624 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:14.480655 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:14.482376 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:14.483150 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:14.483154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:14.483156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:14.483260 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:14.483772 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:14.483776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:14.483777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:14.484027 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:14.484118 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:14.485057 (+ 939us) server_negotiation.cc:299] Negotiation successful
0927 00:47:14.485216 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 00:47:14.487067 4807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:14.470808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35441 (local address 127.0.98.2:42559)
0927 00:47:14.471349 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:14.471397 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:14.472850 (+ 1453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:14.473426 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:14.473434 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:14.473896 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:14.474488 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:14.474503 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.475448 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.475450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:14.476766 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:14.476778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.477348 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.477353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:14.477550 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:14.478353 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:14.478371 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:14.480377 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:14.482508 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:14.482518 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:14.482544 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:14.483000 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:14.483367 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:14.483374 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:14.483376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:14.483629 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:14.484151 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:14.484162 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:14.484698 (+ 536us) client_negotiation.cc:769] Sending connection context
0927 00:47:14.484983 (+ 285us) client_negotiation.cc:240] Negotiation successful
0927 00:47:14.485494 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:47:14.488754 4805 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35441
I20260927 00:47:14.489136 4805 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:14.489825 4805 heartbeater.cc:511] Master 127.0.98.62:35441 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:47:14.491365 4482 ts_manager.cc:194] Registered new tserver with Master: eb324ec8f05f41959fb97a96cbeb11a7 (127.0.98.2:38449)
I20260927 00:47:14.492081 4482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:47:14.509928 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35441
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:14.645860 4810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:14.646268 4810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:14.646322 4810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:14.650954 4810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:14.651077 4810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:14.651157 4810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:47:14.658355 4810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:14.660005 4810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:14.661504 4810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:14.674536 4815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:14.674572 4810 server_base.cc:1034] running on GCE node
W20260927 00:47:14.674536 4818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:14.674597 4816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:14.675383 4810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:14.676324 4810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:14.677533 4810 hybrid_clock.cc:648] HybridClock initialized: now 1790470034677502 us; error 47 us; skew 500 ppm
Sep 27 00:47:14 dist-test-slave-86rs krb5kdc[4438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:14.681386 4810 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:47:14.682494 4810 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:14.682580 4810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:14.682821 4810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:14.685798 4810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-2/data/instance:
uuid: "a8fb5367bcb9440b907f4c064d439490"
format_stamp: "Formatted at 2026-09-27 00:47:14 on dist-test-slave-86rs"
server_key: "6925f18b3dcd0158bcdd5a01dae8544a"
server_key_iv: "16efa71a113aa6719263bfc008e6bddb"
server_key_version: "encryptionkey@0"
I20260927 00:47:14.686553 4810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a8fb5367bcb9440b907f4c064d439490"
format_stamp: "Formatted at 2026-09-27 00:47:14 on dist-test-slave-86rs"
server_key: "6925f18b3dcd0158bcdd5a01dae8544a"
server_key_iv: "16efa71a113aa6719263bfc008e6bddb"
server_key_version: "encryptionkey@0"
I20260927 00:47:14.691761 4810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 00:47:14.695168 4824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:14.697209 4810 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:47:14.697357 4810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "a8fb5367bcb9440b907f4c064d439490"
format_stamp: "Formatted at 2026-09-27 00:47:14 on dist-test-slave-86rs"
server_key: "6925f18b3dcd0158bcdd5a01dae8544a"
server_key_iv: "16efa71a113aa6719263bfc008e6bddb"
server_key_version: "encryptionkey@0"
I20260927 00:47:14.697492 4810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:14.726684 4810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:14.731323 4810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:14.731527 4810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:14.732354 4810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:14.733459 4810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:14.733513 4810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:14.733562 4810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:14.733578 4810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:14.744721 4810 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38321
I20260927 00:47:14.744741 4937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38321 every 8 connection(s)
I20260927 00:47:14.745538 4810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:47:14.748827 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 4810
I20260927 00:47:14.748998 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:47:14.749419 392 external_mini_cluster.cc:1428] Setting key 430fdba117e72b7296f7702bf0c27e60
Sep 27 00:47:14 dist-test-slave-86rs krb5kdc[4438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:14.762480 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:14.748017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34969 (local address 127.0.98.62:35441)
0927 00:47:14.748202 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:14.748206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:14.749504 (+ 1298us) server_negotiation.cc:407] Connection header received
0927 00:47:14.751320 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:14.751324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:14.751396 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:14.751482 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:14.753384 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.754003 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.755302 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.755546 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.755643 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:14.758699 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:14.758702 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:14.758704 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:14.758734 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:14.760240 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:14.760825 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:14.760828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:14.760829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:14.760916 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:14.761258 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:14.761260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:14.761262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:14.761477 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:14.761554 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:14.762170 (+ 616us) server_negotiation.cc:299] Negotiation successful
0927 00:47:14.762310 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 00:47:14.763826 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:14.748600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35441 (local address 127.0.98.3:34969)
0927 00:47:14.749268 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:14.749342 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:14.751063 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:14.751756 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:14.751769 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:14.752473 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:14.753200 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:14.753219 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.754130 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.754134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:14.755170 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:14.755180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.755742 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.755746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:14.755926 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:14.756748 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:14.756769 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:14.758495 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:14.760360 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:14.760365 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:14.760385 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:14.760715 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:14.760997 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:14.760999 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:14.761000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:14.761170 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:14.761561 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:14.761567 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:14.761899 (+ 332us) client_negotiation.cc:769] Sending connection context
0927 00:47:14.762091 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 00:47:14.762452 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":138,"threads_started":1}
I20260927 00:47:14.765380 4938 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35441
I20260927 00:47:14.765702 4938 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:14.766292 4938 heartbeater.cc:511] Master 127.0.98.62:35441 requested a full tablet report, sending...
I20260927 00:47:14.767494 4482 ts_manager.cc:194] Registered new tserver with Master: a8fb5367bcb9440b907f4c064d439490 (127.0.98.3:38321)
I20260927 00:47:14.768177 4482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34969
I20260927 00:47:14.777046 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:47:14.787834 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:14.779522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35146 (local address 127.0.98.62:35441)
0927 00:47:14.779707 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:14.779712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:14.779908 (+ 196us) server_negotiation.cc:407] Connection header received
0927 00:47:14.780148 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:14.780150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:14.780201 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:14.780351 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:14.781285 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.781860 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.782809 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.783031 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.783112 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:14.784637 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:14.784640 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:14.784642 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:14.784674 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:14.786118 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:14.786567 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:14.786571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:14.786572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:14.786657 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:14.786903 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:14.786906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:14.786907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:14.787165 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:14.787283 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:14.787486 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 00:47:14.787604 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:47:14.791373 4482 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:47:14.795042 4482 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:47:14.814131 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:14.806787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38449 (local address 127.0.0.1:45258)
0927 00:47:14.807658 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:14.807671 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:14.807747 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:14.808614 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:14.808621 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:14.808648 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:14.808853 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:14.808862 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.810570 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.810573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:14.812059 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:14.812069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.813236 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.813240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:14.813454 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:14.813532 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 00:47:14.813659 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 00:47:14.813839 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"mutex_wait_us":127,"thread_start_us":127,"threads_started":1}
I20260927 00:47:14.814182 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:14.806885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41512 (local address 127.0.98.3:38321)
0927 00:47:14.807515 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:14.807530 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:14.807873 (+ 343us) server_negotiation.cc:407] Connection header received
0927 00:47:14.807987 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:14.807994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:14.808305 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:14.808511 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:14.808997 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.810374 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.811934 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.812877 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.812993 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:14.813215 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:14.813808 (+ 593us) server_negotiation.cc:299] Negotiation successful
0927 00:47:14.813946 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":193,"threads_started":1}
I20260927 00:47:14.814131 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:14.808210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33997 (local address 127.0.0.1:43512)
0927 00:47:14.808655 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:14.808669 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:14.808768 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:14.809521 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:14.809525 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:14.809545 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:14.809685 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:14.809695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.811465 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.811485 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:14.812551 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:14.812558 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.813655 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.813658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:14.813761 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:14.813785 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:47:14.813836 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 00:47:14.813913 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":1,"thread_start_us":148,"threads_started":1}
I20260927 00:47:14.814648 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:14.807696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45258 (local address 127.0.98.2:38449)
0927 00:47:14.808090 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:14.808108 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:14.808133 (+ 25us) server_negotiation.cc:407] Connection header received
0927 00:47:14.808210 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:14.808215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:14.808417 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:14.809032 (+ 615us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:14.809244 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.810390 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.812254 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.813093 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.814076 (+ 983us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:14.814309 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:14.814372 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 00:47:14.814501 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":149,"threads_started":1}
I20260927 00:47:14.815095 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:14.808448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43512 (local address 127.0.98.1:33997)
0927 00:47:14.809026 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:14.809033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:14.809060 (+ 27us) server_negotiation.cc:407] Connection header received
0927 00:47:14.809149 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:14.809153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:14.809321 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:14.809862 (+ 541us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:14.810024 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.811248 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.812675 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.813512 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.814591 (+ 1079us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:14.814775 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:14.814826 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 00:47:14.814931 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":114,"threads_started":1}
I20260927 00:47:14.815850 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:14.806731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38321 (local address 127.0.0.1:41512)
0927 00:47:14.807436 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:14.807504 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:14.807700 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:14.808756 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:14.808759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:14.808772 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:14.808867 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:14.808872 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.810556 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.810563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:14.811789 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:14.811799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.813265 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.813267 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:14.813500 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:14.813532 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 00:47:14.813659 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 00:47:14.813992 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":303,"thread_start_us":133,"threads_started":1}
I20260927 00:47:14.818558 4606 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed2980d0f834fae90834f66ce1bbace (DEFAULT_TABLE table=test-table [id=02a0194104524dc68daae13b8e0e3103]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:14.818553 4739 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed2980d0f834fae90834f66ce1bbace (DEFAULT_TABLE table=test-table [id=02a0194104524dc68daae13b8e0e3103]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:14.818660 4872 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed2980d0f834fae90834f66ce1bbace (DEFAULT_TABLE table=test-table [id=02a0194104524dc68daae13b8e0e3103]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:14.819818 4739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed2980d0f834fae90834f66ce1bbace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:14.819828 4872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed2980d0f834fae90834f66ce1bbace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:14.819825 4606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed2980d0f834fae90834f66ce1bbace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:14.828763 4955 tablet_bootstrap.cc:492] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7: Bootstrap starting.
I20260927 00:47:14.831308 4955 tablet_bootstrap.cc:654] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:14.832388 4956 tablet_bootstrap.cc:492] T 4ed2980d0f834fae90834f66ce1bbace P 09b07b7a3434483b9ca37871b266432a: Bootstrap starting.
I20260927 00:47:14.832649 4955 log.cc:824] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:14.833518 4957 tablet_bootstrap.cc:492] T 4ed2980d0f834fae90834f66ce1bbace P a8fb5367bcb9440b907f4c064d439490: Bootstrap starting.
I20260927 00:47:14.835264 4956 tablet_bootstrap.cc:654] T 4ed2980d0f834fae90834f66ce1bbace P 09b07b7a3434483b9ca37871b266432a: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:14.835378 4955 tablet_bootstrap.cc:492] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7: No bootstrap required, opened a new log
I20260927 00:47:14.835670 4955 ts_tablet_manager.cc:1397] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 00:47:14.835728 4957 tablet_bootstrap.cc:654] T 4ed2980d0f834fae90834f66ce1bbace P a8fb5367bcb9440b907f4c064d439490: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:14.836320 4956 log.cc:824] T 4ed2980d0f834fae90834f66ce1bbace P 09b07b7a3434483b9ca37871b266432a: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:14.837293 4957 log.cc:824] T 4ed2980d0f834fae90834f66ce1bbace P a8fb5367bcb9440b907f4c064d439490: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:14.838752 4956 tablet_bootstrap.cc:492] T 4ed2980d0f834fae90834f66ce1bbace P 09b07b7a3434483b9ca37871b266432a: No bootstrap required, opened a new log
I20260927 00:47:14.838984 4956 ts_tablet_manager.cc:1397] T 4ed2980d0f834fae90834f66ce1bbace P 09b07b7a3434483b9ca37871b266432a: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260927 00:47:14.839504 4957 tablet_bootstrap.cc:492] T 4ed2980d0f834fae90834f66ce1bbace P a8fb5367bcb9440b907f4c064d439490: No bootstrap required, opened a new log
I20260927 00:47:14.839711 4957 ts_tablet_manager.cc:1397] T 4ed2980d0f834fae90834f66ce1bbace P a8fb5367bcb9440b907f4c064d439490: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:47:14.839926 4955 raft_consensus.cc:348] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fb5367bcb9440b907f4c064d439490" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38321 } } peers { permanent_uuid: "eb324ec8f05f41959fb97a96cbeb11a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38449 } } peers { permanent_uuid: "09b07b7a3434483b9ca37871b266432a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33997 } }
I20260927 00:47:14.840229 4955 raft_consensus.cc:374] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:14.840327 4955 raft_consensus.cc:729] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb324ec8f05f41959fb97a96cbeb11a7, State: Initialized, Role: FOLLOWER
I20260927 00:47:14.840901 4955 consensus_queue.cc:260] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fb5367bcb9440b907f4c064d439490" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38321 } } peers { permanent_uuid: "eb324ec8f05f41959fb97a96cbeb11a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38449 } } peers { permanent_uuid: "09b07b7a3434483b9ca37871b266432a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33997 } }
I20260927 00:47:14.841775 4805 heartbeater.cc:503] Master 127.0.98.62:35441 was elected leader, sending a full tablet report...
I20260927 00:47:14.842176 4955 ts_tablet_manager.cc:1428] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260927 00:47:14.844467 4956 raft_consensus.cc:348] T 4ed2980d0f834fae90834f66ce1bbace P 09b07b7a3434483b9ca37871b266432a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fb5367bcb9440b907f4c064d439490" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38321 } } peers { permanent_uuid: "eb324ec8f05f41959fb97a96cbeb11a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38449 } } peers { permanent_uuid: "09b07b7a3434483b9ca37871b266432a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33997 } }
I20260927 00:47:14.845006 4956 raft_consensus.cc:374] T 4ed2980d0f834fae90834f66ce1bbace P 09b07b7a3434483b9ca37871b266432a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:14.845144 4956 raft_consensus.cc:729] T 4ed2980d0f834fae90834f66ce1bbace P 09b07b7a3434483b9ca37871b266432a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09b07b7a3434483b9ca37871b266432a, State: Initialized, Role: FOLLOWER
I20260927 00:47:14.845184 4957 raft_consensus.cc:348] T 4ed2980d0f834fae90834f66ce1bbace P a8fb5367bcb9440b907f4c064d439490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fb5367bcb9440b907f4c064d439490" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38321 } } peers { permanent_uuid: "eb324ec8f05f41959fb97a96cbeb11a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38449 } } peers { permanent_uuid: "09b07b7a3434483b9ca37871b266432a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33997 } }
I20260927 00:47:14.845582 4957 raft_consensus.cc:374] T 4ed2980d0f834fae90834f66ce1bbace P a8fb5367bcb9440b907f4c064d439490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:14.845676 4957 raft_consensus.cc:729] T 4ed2980d0f834fae90834f66ce1bbace P a8fb5367bcb9440b907f4c064d439490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8fb5367bcb9440b907f4c064d439490, State: Initialized, Role: FOLLOWER
I20260927 00:47:14.845875 4956 consensus_queue.cc:260] T 4ed2980d0f834fae90834f66ce1bbace P 09b07b7a3434483b9ca37871b266432a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fb5367bcb9440b907f4c064d439490" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38321 } } peers { permanent_uuid: "eb324ec8f05f41959fb97a96cbeb11a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38449 } } peers { permanent_uuid: "09b07b7a3434483b9ca37871b266432a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33997 } }
I20260927 00:47:14.846302 4957 consensus_queue.cc:260] T 4ed2980d0f834fae90834f66ce1bbace P a8fb5367bcb9440b907f4c064d439490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fb5367bcb9440b907f4c064d439490" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38321 } } peers { permanent_uuid: "eb324ec8f05f41959fb97a96cbeb11a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38449 } } peers { permanent_uuid: "09b07b7a3434483b9ca37871b266432a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33997 } }
I20260927 00:47:14.846814 4672 heartbeater.cc:503] Master 127.0.98.62:35441 was elected leader, sending a full tablet report...
I20260927 00:47:14.847335 4956 ts_tablet_manager.cc:1428] T 4ed2980d0f834fae90834f66ce1bbace P 09b07b7a3434483b9ca37871b266432a: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260927 00:47:14.847335 4957 ts_tablet_manager.cc:1428] T 4ed2980d0f834fae90834f66ce1bbace P a8fb5367bcb9440b907f4c064d439490: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260927 00:47:14.847774 4938 heartbeater.cc:503] Master 127.0.98.62:35441 was elected leader, sending a full tablet report...
I20260927 00:47:14.857707 4961 raft_consensus.cc:482] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:47:14.857965 4961 raft_consensus.cc:504] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fb5367bcb9440b907f4c064d439490" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38321 } } peers { permanent_uuid: "eb324ec8f05f41959fb97a96cbeb11a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38449 } } peers { permanent_uuid: "09b07b7a3434483b9ca37871b266432a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33997 } }
I20260927 00:47:14.859447 4961 leader_election.cc:288] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8fb5367bcb9440b907f4c064d439490 (127.0.98.3:38321), 09b07b7a3434483b9ca37871b266432a (127.0.98.1:33997)
I20260927 00:47:14.865003 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:14.860041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43509 (local address 127.0.98.3:38321)
0927 00:47:14.860197 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:14.860204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:14.860268 (+ 64us) server_negotiation.cc:407] Connection header received
0927 00:47:14.860337 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:14.860341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:14.860408 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:14.860508 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:14.860844 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.861704 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.863647 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.864296 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.864411 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:14.864476 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:14.864799 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 00:47:14.864874 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:47:14.865028 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:14.859867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56659 (local address 127.0.98.1:33997)
0927 00:47:14.860004 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:14.860011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:14.860597 (+ 586us) server_negotiation.cc:407] Connection header received
0927 00:47:14.860798 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:14.860801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:14.860871 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:14.860955 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:14.861233 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.861735 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.863662 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.864181 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.864311 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:14.864405 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:14.864807 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 00:47:14.864897 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:47:14.865028 4807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:14.859708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38321 (local address 127.0.98.2:43509)
0927 00:47:14.859838 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:14.859856 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:14.859958 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:14.860522 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:14.860525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:14.860545 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:14.860698 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:14.860704 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.861831 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.861835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:14.863482 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:14.863495 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.864423 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.864429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:14.864589 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:14.864688 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 00:47:14.864778 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 00:47:14.864875 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":188}
I20260927 00:47:14.865470 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:14.859707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33997 (local address 127.0.98.2:56659)
0927 00:47:14.860488 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:14.860503 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:14.860668 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:14.860974 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:14.860977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:14.860992 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:14.861127 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:14.861133 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.861877 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.861879 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:14.863482 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:14.863495 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.864489 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.864492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:14.864680 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:14.864698 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 00:47:14.865178 (+ 480us) client_negotiation.cc:240] Negotiation successful
0927 00:47:14.865275 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"mutex_wait_us":514,"thread_start_us":206,"threads_started":1}
I20260927 00:47:14.865870 4892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed2980d0f834fae90834f66ce1bbace" candidate_uuid: "eb324ec8f05f41959fb97a96cbeb11a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8fb5367bcb9440b907f4c064d439490" is_pre_election: true
I20260927 00:47:14.866209 4892 raft_consensus.cc:2454] T 4ed2980d0f834fae90834f66ce1bbace P a8fb5367bcb9440b907f4c064d439490 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb324ec8f05f41959fb97a96cbeb11a7 in term 0.
I20260927 00:47:14.866822 4694 leader_election.cc:302] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8fb5367bcb9440b907f4c064d439490, eb324ec8f05f41959fb97a96cbeb11a7; no voters:
I20260927 00:47:14.867058 4626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed2980d0f834fae90834f66ce1bbace" candidate_uuid: "eb324ec8f05f41959fb97a96cbeb11a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09b07b7a3434483b9ca37871b266432a" is_pre_election: true
I20260927 00:47:14.867201 4961 raft_consensus.cc:2781] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:47:14.867307 4961 raft_consensus.cc:482] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:47:14.867363 4961 raft_consensus.cc:3037] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:14.867455 4626 raft_consensus.cc:2454] T 4ed2980d0f834fae90834f66ce1bbace P 09b07b7a3434483b9ca37871b266432a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb324ec8f05f41959fb97a96cbeb11a7 in term 0.
I20260927 00:47:14.868992 4961 raft_consensus.cc:504] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fb5367bcb9440b907f4c064d439490" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38321 } } peers { permanent_uuid: "eb324ec8f05f41959fb97a96cbeb11a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38449 } } peers { permanent_uuid: "09b07b7a3434483b9ca37871b266432a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33997 } }
I20260927 00:47:14.869477 4961 leader_election.cc:288] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7 [CANDIDATE]: Term 1 election: Requested vote from peers a8fb5367bcb9440b907f4c064d439490 (127.0.98.3:38321), 09b07b7a3434483b9ca37871b266432a (127.0.98.1:33997)
I20260927 00:47:14.869830 4892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed2980d0f834fae90834f66ce1bbace" candidate_uuid: "eb324ec8f05f41959fb97a96cbeb11a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8fb5367bcb9440b907f4c064d439490"
I20260927 00:47:14.869874 4626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed2980d0f834fae90834f66ce1bbace" candidate_uuid: "eb324ec8f05f41959fb97a96cbeb11a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09b07b7a3434483b9ca37871b266432a"
I20260927 00:47:14.869977 4892 raft_consensus.cc:3037] T 4ed2980d0f834fae90834f66ce1bbace P a8fb5367bcb9440b907f4c064d439490 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:14.870039 4626 raft_consensus.cc:3037] T 4ed2980d0f834fae90834f66ce1bbace P 09b07b7a3434483b9ca37871b266432a [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:14.871227 4626 raft_consensus.cc:2454] T 4ed2980d0f834fae90834f66ce1bbace P 09b07b7a3434483b9ca37871b266432a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb324ec8f05f41959fb97a96cbeb11a7 in term 1.
I20260927 00:47:14.871228 4892 raft_consensus.cc:2454] T 4ed2980d0f834fae90834f66ce1bbace P a8fb5367bcb9440b907f4c064d439490 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb324ec8f05f41959fb97a96cbeb11a7 in term 1.
I20260927 00:47:14.871578 4692 leader_election.cc:302] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09b07b7a3434483b9ca37871b266432a, eb324ec8f05f41959fb97a96cbeb11a7; no voters:
I20260927 00:47:14.871768 4961 raft_consensus.cc:2781] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:14.872046 4961 raft_consensus.cc:686] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7 [term 1 LEADER]: Becoming Leader. State: Replica: eb324ec8f05f41959fb97a96cbeb11a7, State: Running, Role: LEADER
I20260927 00:47:14.872522 4961 consensus_queue.cc:237] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fb5367bcb9440b907f4c064d439490" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38321 } } peers { permanent_uuid: "eb324ec8f05f41959fb97a96cbeb11a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38449 } } peers { permanent_uuid: "09b07b7a3434483b9ca37871b266432a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33997 } }
I20260927 00:47:14.876734 4482 catalog_manager.cc:5495] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb324ec8f05f41959fb97a96cbeb11a7 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "eb324ec8f05f41959fb97a96cbeb11a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fb5367bcb9440b907f4c064d439490" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb324ec8f05f41959fb97a96cbeb11a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09b07b7a3434483b9ca37871b266432a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33997 } health_report { overall_health: UNKNOWN } } }
I20260927 00:47:14.895394 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:14.891077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45274 (local address 127.0.98.2:38449)
0927 00:47:14.891235 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:14.891240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:14.891257 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:47:14.891419 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:14.891422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:14.891489 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:14.891654 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:47:14.891942 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.892630 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.893567 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:14.893822 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:14.893946 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:14.894307 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:47:14.894903 (+ 596us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:47:14.895065 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:14.895098 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 00:47:14.895176 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:47:14.904057 4892 raft_consensus.cc:1261] T 4ed2980d0f834fae90834f66ce1bbace P a8fb5367bcb9440b907f4c064d439490 [term 1 FOLLOWER]: Refusing update from remote peer eb324ec8f05f41959fb97a96cbeb11a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:47:14.904057 4626 raft_consensus.cc:1261] T 4ed2980d0f834fae90834f66ce1bbace P 09b07b7a3434483b9ca37871b266432a [term 1 FOLLOWER]: Refusing update from remote peer eb324ec8f05f41959fb97a96cbeb11a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:47:14.905100 4965 consensus_queue.cc:1035] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09b07b7a3434483b9ca37871b266432a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:47:14.905423 4961 consensus_queue.cc:1035] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8fb5367bcb9440b907f4c064d439490" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:47:14.928630 4482 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:47:14.928918 4482 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:47:14.932170 4482 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 352bee5d05c34231ba78730b62243a77: Sending DeleteTablet for 3 replicas of tablet 4ed2980d0f834fae90834f66ce1bbace
I20260927 00:47:14.933364 4872 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ed2980d0f834fae90834f66ce1bbace with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41512
I20260927 00:47:14.933454 4606 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ed2980d0f834fae90834f66ce1bbace with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43512
I20260927 00:47:14.934063 4739 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ed2980d0f834fae90834f66ce1bbace with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45258
I20260927 00:47:14.934211 4978 tablet_replica.cc:330] T 4ed2980d0f834fae90834f66ce1bbace P 09b07b7a3434483b9ca37871b266432a: stopping tablet replica
I20260927 00:47:14.934481 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 4544
I20260927 00:47:14.935105 4979 tablet_replica.cc:330] T 4ed2980d0f834fae90834f66ce1bbace P a8fb5367bcb9440b907f4c064d439490: stopping tablet replica
I20260927 00:47:14.935530 4979 raft_consensus.cc:2229] T 4ed2980d0f834fae90834f66ce1bbace P a8fb5367bcb9440b907f4c064d439490 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:47:14.935964 4979 raft_consensus.cc:2258] T 4ed2980d0f834fae90834f66ce1bbace P a8fb5367bcb9440b907f4c064d439490 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:14.937772 4980 tablet_replica.cc:330] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7: stopping tablet replica
I20260927 00:47:14.938237 4980 raft_consensus.cc:2229] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:47:14.938820 4980 raft_consensus.cc:2258] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:14.939232 4979 ts_tablet_manager.cc:1905] T 4ed2980d0f834fae90834f66ce1bbace P a8fb5367bcb9440b907f4c064d439490: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:14.940764 4980 ts_tablet_manager.cc:1905] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:14.942729 4979 ts_tablet_manager.cc:1918] T 4ed2980d0f834fae90834f66ce1bbace P a8fb5367bcb9440b907f4c064d439490: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:47:14.942860 4979 log.cc:1196] T 4ed2980d0f834fae90834f66ce1bbace P a8fb5367bcb9440b907f4c064d439490: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-2/wal/wals/4ed2980d0f834fae90834f66ce1bbace
I20260927 00:47:14.943238 4979 ts_tablet_manager.cc:1939] T 4ed2980d0f834fae90834f66ce1bbace P a8fb5367bcb9440b907f4c064d439490: Deleting consensus metadata
W20260927 00:47:14.943650 4466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33997 (error 108)
I20260927 00:47:14.944427 4980 ts_tablet_manager.cc:1918] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:47:14.944636 4980 log.cc:1196] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790470006085796-392-0/minicluster-data/ts-1/wal/wals/4ed2980d0f834fae90834f66ce1bbace
I20260927 00:47:14.945062 4980 ts_tablet_manager.cc:1939] T 4ed2980d0f834fae90834f66ce1bbace P eb324ec8f05f41959fb97a96cbeb11a7: Deleting consensus metadata
I20260927 00:47:14.945175 4468 catalog_manager.cc:4841] TS a8fb5367bcb9440b907f4c064d439490 (127.0.98.3:38321): tablet 4ed2980d0f834fae90834f66ce1bbace (table test-table [id=02a0194104524dc68daae13b8e0e3103]) successfully deleted
I20260927 00:47:14.945499 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 4677
I20260927 00:47:14.945549 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:14.945194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33997 (local address 127.0.0.1:43528)
0927 00:47:14.945343 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:14.945431 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 00:47:14.946357 4466 catalog_manager.cc:4584] TS 09b07b7a3434483b9ca37871b266432a (127.0.98.1:33997): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4ed2980d0f834fae90834f66ce1bbace: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33997: connect: Connection refused (error 111)
I20260927 00:47:14.953320 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 4810
I20260927 00:47:14.955121 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:14.953635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38449 (local address 127.0.0.1:45278)
0927 00:47:14.954885 (+ 1250us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:14.954972 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1111}
I20260927 00:47:14.960374 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 4451
2026-09-27T00:47:14Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2627 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T00:47:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:47:14Z Disabled control of system clock
I20260927 00:47:14.997064 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43835
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 00:47:15.154767 4985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:15.155184 4985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:15.155227 4985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:15.160004 4985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:47:15.160130 4985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:15.160148 4985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:15.160168 4985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:47:15.160184 4985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:47:15.166716 4985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43835
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:15.168408 4985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:15.170361 4985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:47:15.181711 4985 server_base.cc:1034] running on GCE node
W20260927 00:47:15.181634 4990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:15.181676 4991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:15.181656 4993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:15.182766 4985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:15.184629 4985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:15.185877 4985 hybrid_clock.cc:648] HybridClock initialized: now 1790470035185813 us; error 89 us; skew 500 ppm
I20260927 00:47:15.189301 4985 webserver.cc:460] Webserver started at http://127.0.98.62:44809/ using document root <none> and password file <none>
I20260927 00:47:15.190224 4985 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:15.190318 4985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:15.190734 4985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:15.194124 4985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "41bfecc9d1854a778eb088372d0ae3a7"
format_stamp: "Formatted at 2026-09-27 00:47:15 on dist-test-slave-86rs"
server_key: "673df74fc3700edd407caa73fd6ce135"
server_key_iv: "613a785c80141f2d90092894678db847"
server_key_version: "encryptionkey@0"
I20260927 00:47:15.195029 4985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "41bfecc9d1854a778eb088372d0ae3a7"
format_stamp: "Formatted at 2026-09-27 00:47:15 on dist-test-slave-86rs"
server_key: "673df74fc3700edd407caa73fd6ce135"
server_key_iv: "613a785c80141f2d90092894678db847"
server_key_version: "encryptionkey@0"
I20260927 00:47:15.200775 4985 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260927 00:47:15.204373 4999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:15.206266 4985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:47:15.206471 4985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "41bfecc9d1854a778eb088372d0ae3a7"
format_stamp: "Formatted at 2026-09-27 00:47:15 on dist-test-slave-86rs"
server_key: "673df74fc3700edd407caa73fd6ce135"
server_key_iv: "613a785c80141f2d90092894678db847"
server_key_version: "encryptionkey@0"
I20260927 00:47:15.206673 4985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:15.212719 4985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:15.213603 4985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:15.213827 4985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:15.222760 4985 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43835
I20260927 00:47:15.222765 5051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43835 every 8 connection(s)
I20260927 00:47:15.224125 4985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:47:15.225842 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 4985
I20260927 00:47:15.226008 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:47:15.226368 392 external_mini_cluster.cc:1428] Setting key 4d17dd65e95a24f76a568059d746cb1f
I20260927 00:47:15.228861 5052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:15.233374 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:15.228188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37234 (local address 127.0.98.62:43835)
0927 00:47:15.229077 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:15.229096 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:15.229146 (+ 50us) server_negotiation.cc:407] Connection header received
0927 00:47:15.229931 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:15.229952 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:15.230428 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:15.230786 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:15.231734 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:15.231740 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:15.231754 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:47:15.231764 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:15.231825 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:47:15.231843 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:15.231967 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:15.232112 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 00:47:15.232467 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":152,"threads_started":1}
I20260927 00:47:15.236794 5052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41bfecc9d1854a778eb088372d0ae3a7: Bootstrap starting.
I20260927 00:47:15.239667 5052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41bfecc9d1854a778eb088372d0ae3a7: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:15.240823 5052 log.cc:824] T 00000000000000000000000000000000 P 41bfecc9d1854a778eb088372d0ae3a7: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:15.243291 5052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41bfecc9d1854a778eb088372d0ae3a7: No bootstrap required, opened a new log
I20260927 00:47:15.247601 5052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41bfecc9d1854a778eb088372d0ae3a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bfecc9d1854a778eb088372d0ae3a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43835 } }
I20260927 00:47:15.247956 5052 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41bfecc9d1854a778eb088372d0ae3a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:15.248039 5052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41bfecc9d1854a778eb088372d0ae3a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41bfecc9d1854a778eb088372d0ae3a7, State: Initialized, Role: FOLLOWER
I20260927 00:47:15.248657 5052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41bfecc9d1854a778eb088372d0ae3a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bfecc9d1854a778eb088372d0ae3a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43835 } }
I20260927 00:47:15.248833 5052 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41bfecc9d1854a778eb088372d0ae3a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:47:15.248925 5052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41bfecc9d1854a778eb088372d0ae3a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:47:15.249039 5052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41bfecc9d1854a778eb088372d0ae3a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:15.250178 5052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41bfecc9d1854a778eb088372d0ae3a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bfecc9d1854a778eb088372d0ae3a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43835 } }
I20260927 00:47:15.250612 5052 leader_election.cc:302] T 00000000000000000000000000000000 P 41bfecc9d1854a778eb088372d0ae3a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41bfecc9d1854a778eb088372d0ae3a7; no voters:
I20260927 00:47:15.251030 5052 leader_election.cc:288] T 00000000000000000000000000000000 P 41bfecc9d1854a778eb088372d0ae3a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:47:15.251178 5057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41bfecc9d1854a778eb088372d0ae3a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:15.251453 5057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41bfecc9d1854a778eb088372d0ae3a7 [term 1 LEADER]: Becoming Leader. State: Replica: 41bfecc9d1854a778eb088372d0ae3a7, State: Running, Role: LEADER
I20260927 00:47:15.251827 5057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41bfecc9d1854a778eb088372d0ae3a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bfecc9d1854a778eb088372d0ae3a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43835 } }
I20260927 00:47:15.252666 5052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41bfecc9d1854a778eb088372d0ae3a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:47:15.253898 5059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41bfecc9d1854a778eb088372d0ae3a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41bfecc9d1854a778eb088372d0ae3a7. Latest consensus state: current_term: 1 leader_uuid: "41bfecc9d1854a778eb088372d0ae3a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bfecc9d1854a778eb088372d0ae3a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43835 } } }
I20260927 00:47:15.254031 5059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41bfecc9d1854a778eb088372d0ae3a7 [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:15.254323 5058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41bfecc9d1854a778eb088372d0ae3a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41bfecc9d1854a778eb088372d0ae3a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bfecc9d1854a778eb088372d0ae3a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43835 } } }
I20260927 00:47:15.254426 5058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41bfecc9d1854a778eb088372d0ae3a7 [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:15.254540 5066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:47:15.258081 5066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:47:15.266233 5066 catalog_manager.cc:1347] Generated new cluster ID: 3493510c59ed4fdab5e9ced59e901f8f
I20260927 00:47:15.266352 5066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:47:15.298370 5066 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:47:15.298874 5066 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:47:15.306478 5066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41bfecc9d1854a778eb088372d0ae3a7: Generated new TSK 0
I20260927 00:47:15.307439 5066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:47:15.318989 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43835
--builtin_ntp_servers=127.0.98.20:33215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 00:47:15.459429 5076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:15.459795 5076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:15.459843 5076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:15.464310 5076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:15.464407 5076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:15.464475 5076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:47:15.469805 5076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:15.471360 5076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:15.473367 5076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:15.485800 5084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:15.485826 5081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:15.485827 5082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:15.486129 5076 server_base.cc:1034] running on GCE node
I20260927 00:47:15.486860 5076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:15.487673 5076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:15.488857 5076 hybrid_clock.cc:648] HybridClock initialized: now 1790470035488819 us; error 54 us; skew 500 ppm
I20260927 00:47:15.490976 5076 webserver.cc:460] Webserver started at http://127.0.98.1:44167/ using document root <none> and password file <none>
I20260927 00:47:15.491696 5076 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:15.491793 5076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:15.492069 5076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:15.494735 5076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-0/data/instance:
uuid: "42c050bcb1e24c5197bceb4457ff2687"
format_stamp: "Formatted at 2026-09-27 00:47:15 on dist-test-slave-86rs"
server_key: "a24e6605ac32ebb7960ab921857ab52b"
server_key_iv: "edfc5ab453c7722c9e78c656bba52f5c"
server_key_version: "encryptionkey@0"
I20260927 00:47:15.495409 5076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-0/wal/instance:
uuid: "42c050bcb1e24c5197bceb4457ff2687"
format_stamp: "Formatted at 2026-09-27 00:47:15 on dist-test-slave-86rs"
server_key: "a24e6605ac32ebb7960ab921857ab52b"
server_key_iv: "edfc5ab453c7722c9e78c656bba52f5c"
server_key_version: "encryptionkey@0"
I20260927 00:47:15.500145 5076 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 00:47:15.503219 5090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:15.504813 5076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 00:47:15.504982 5076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "42c050bcb1e24c5197bceb4457ff2687"
format_stamp: "Formatted at 2026-09-27 00:47:15 on dist-test-slave-86rs"
server_key: "a24e6605ac32ebb7960ab921857ab52b"
server_key_iv: "edfc5ab453c7722c9e78c656bba52f5c"
server_key_version: "encryptionkey@0"
I20260927 00:47:15.505167 5076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:15.510772 5076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:15.511606 5076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:15.511809 5076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:15.512631 5076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:15.513837 5076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:15.513912 5076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:15.513981 5076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:15.514035 5076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:15.525547 5076 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35159
I20260927 00:47:15.525573 5203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35159 every 8 connection(s)
I20260927 00:47:15.526921 5076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:47:15.527926 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 5076
I20260927 00:47:15.528100 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:47:15.528528 392 external_mini_cluster.cc:1428] Setting key 88644c2f8618c19dbc20930baf509f01
I20260927 00:47:15.531574 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43835
--builtin_ntp_servers=127.0.98.20:33215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 00:47:15.536589 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:15.529953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44171 (local address 127.0.98.62:43835)
0927 00:47:15.530120 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:15.530123 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:15.532198 (+ 2075us) server_negotiation.cc:407] Connection header received
0927 00:47:15.534023 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:15.534029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:15.534099 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:15.534233 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:15.535894 (+ 1661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:15.535898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:15.535901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:47:15.535920 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:15.535956 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:47:15.535967 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:15.536103 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:15.536307 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 00:47:15.536380 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:47:15.538021 5206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:15.530796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.1:44171)
0927 00:47:15.531984 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:15.532056 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:15.533678 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:15.534507 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:15.534518 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:15.534663 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:15.535498 (+ 835us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:47:15.535499 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:15.535561 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:47:15.535670 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:15.536071 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:15.536083 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:15.536091 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 00:47:15.536382 (+ 291us) client_negotiation.cc:240] Negotiation successful
0927 00:47:15.536904 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":191,"threads_started":1}
I20260927 00:47:15.539899 5204 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43835
I20260927 00:47:15.540356 5204 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:15.541138 5204 heartbeater.cc:511] Master 127.0.98.62:43835 requested a full tablet report, sending...
I20260927 00:47:15.543774 5016 ts_manager.cc:194] Registered new tserver with Master: 42c050bcb1e24c5197bceb4457ff2687 (127.0.98.1:35159)
W20260927 00:47:15.669740 5207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:15.670300 5207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:15.670408 5207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:15.675158 5207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:15.675246 5207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:15.675314 5207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:47:15.680776 5207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:15.682479 5207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:15.684365 5207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:15.696216 5212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:15.696348 5215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:15.696482 5207 server_base.cc:1034] running on GCE node
W20260927 00:47:15.696241 5213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:15.697091 5207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:15.697906 5207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:15.699090 5207 hybrid_clock.cc:648] HybridClock initialized: now 1790470035699071 us; error 31 us; skew 500 ppm
I20260927 00:47:15.701349 5207 webserver.cc:460] Webserver started at http://127.0.98.2:36919/ using document root <none> and password file <none>
I20260927 00:47:15.702077 5207 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:15.702147 5207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:15.702435 5207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:15.705068 5207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-1/data/instance:
uuid: "ec7742562bfa4f88a671e738bd983b56"
format_stamp: "Formatted at 2026-09-27 00:47:15 on dist-test-slave-86rs"
server_key: "2c80fec7a8b34415263b82951e6ba7be"
server_key_iv: "60b7ff53bcecf1c749e95c6df9c9cbb6"
server_key_version: "encryptionkey@0"
I20260927 00:47:15.705734 5207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ec7742562bfa4f88a671e738bd983b56"
format_stamp: "Formatted at 2026-09-27 00:47:15 on dist-test-slave-86rs"
server_key: "2c80fec7a8b34415263b82951e6ba7be"
server_key_iv: "60b7ff53bcecf1c749e95c6df9c9cbb6"
server_key_version: "encryptionkey@0"
I20260927 00:47:15.710556 5207 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 00:47:15.713835 5221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:15.715451 5207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:47:15.715615 5207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "ec7742562bfa4f88a671e738bd983b56"
format_stamp: "Formatted at 2026-09-27 00:47:15 on dist-test-slave-86rs"
server_key: "2c80fec7a8b34415263b82951e6ba7be"
server_key_iv: "60b7ff53bcecf1c749e95c6df9c9cbb6"
server_key_version: "encryptionkey@0"
I20260927 00:47:15.715754 5207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:15.721710 5207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:15.722494 5207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:15.722711 5207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:15.723387 5207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:15.724597 5207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:15.724646 5207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:15.724689 5207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:15.724742 5207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:15.734858 5207 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42989
I20260927 00:47:15.734884 5334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42989 every 8 connection(s)
I20260927 00:47:15.736107 5207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:47:15.740409 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 5207
I20260927 00:47:15.740602 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:47:15.740969 392 external_mini_cluster.cc:1428] Setting key 06aad4ed82996e3f0c11a8bf34418d94
I20260927 00:47:15.743650 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43835
--builtin_ntp_servers=127.0.98.20:33215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 00:47:15.744678 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:15.738674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51903 (local address 127.0.98.62:43835)
0927 00:47:15.738838 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:15.738842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:15.740471 (+ 1629us) server_negotiation.cc:407] Connection header received
0927 00:47:15.742372 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:15.742376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:15.742418 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:15.742518 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:15.744080 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:15.744084 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:15.744086 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:47:15.744098 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:15.744129 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:47:15.744138 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:15.744286 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:15.744491 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 00:47:15.744544 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:47:15.746166 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:15.739162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.2:51903)
0927 00:47:15.740196 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:15.740300 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:15.742120 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:15.742791 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:15.742802 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:15.742957 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:15.743697 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:47:15.743699 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:15.743770 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:47:15.743897 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:15.744265 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:15.744277 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:15.744287 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 00:47:15.744508 (+ 221us) client_negotiation.cc:240] Negotiation successful
0927 00:47:15.744925 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":144,"threads_started":1}
I20260927 00:47:15.747740 5335 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43835
I20260927 00:47:15.748046 5335 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:15.748682 5335 heartbeater.cc:511] Master 127.0.98.62:43835 requested a full tablet report, sending...
I20260927 00:47:15.750192 5016 ts_manager.cc:194] Registered new tserver with Master: ec7742562bfa4f88a671e738bd983b56 (127.0.98.2:42989)
W20260927 00:47:15.882010 5338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:15.882380 5338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:15.882429 5338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:15.886778 5338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:15.886864 5338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:15.886932 5338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:47:15.892185 5338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:15.893707 5338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:15.895542 5338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:15.907560 5343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:15.907750 5338 server_base.cc:1034] running on GCE node
W20260927 00:47:15.907728 5344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:15.907987 5346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:15.908609 5338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:15.909526 5338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:15.910760 5338 hybrid_clock.cc:648] HybridClock initialized: now 1790470035910729 us; error 47 us; skew 500 ppm
I20260927 00:47:15.913295 5338 webserver.cc:460] Webserver started at http://127.0.98.3:45507/ using document root <none> and password file <none>
I20260927 00:47:15.914132 5338 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:15.914237 5338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:15.914520 5338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:15.917330 5338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-2/data/instance:
uuid: "bfaa1283912c4692a772ab1c175028fa"
format_stamp: "Formatted at 2026-09-27 00:47:15 on dist-test-slave-86rs"
server_key: "9868c61bff5c26543f93655384642ee9"
server_key_iv: "38e397a0bdd328a5a3a454a5d8d219b8"
server_key_version: "encryptionkey@0"
I20260927 00:47:15.918061 5338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bfaa1283912c4692a772ab1c175028fa"
format_stamp: "Formatted at 2026-09-27 00:47:15 on dist-test-slave-86rs"
server_key: "9868c61bff5c26543f93655384642ee9"
server_key_iv: "38e397a0bdd328a5a3a454a5d8d219b8"
server_key_version: "encryptionkey@0"
I20260927 00:47:15.923030 5338 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 00:47:15.926177 5352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:15.927963 5338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 00:47:15.928169 5338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "bfaa1283912c4692a772ab1c175028fa"
format_stamp: "Formatted at 2026-09-27 00:47:15 on dist-test-slave-86rs"
server_key: "9868c61bff5c26543f93655384642ee9"
server_key_iv: "38e397a0bdd328a5a3a454a5d8d219b8"
server_key_version: "encryptionkey@0"
I20260927 00:47:15.928390 5338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:15.935825 5338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:15.937004 5338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:15.937244 5338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:15.938295 5338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:15.939937 5338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:15.940011 5338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:15.940097 5338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:15.940141 5338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:15.951779 5338 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41979
I20260927 00:47:15.951807 5465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41979 every 8 connection(s)
I20260927 00:47:15.953133 5338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:47:15.962900 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 5338
I20260927 00:47:15.962966 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:15.955789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47375 (local address 127.0.98.62:43835)
0927 00:47:15.955991 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:15.955997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:15.957766 (+ 1769us) server_negotiation.cc:407] Connection header received
0927 00:47:15.959653 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:15.959657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:15.959716 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:15.959811 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:15.962294 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:15.962298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:15.962301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:47:15.962319 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:15.962348 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:47:15.962360 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:15.962464 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:15.962699 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 00:47:15.962765 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 00:47:15.963064 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:47:15.963423 392 external_mini_cluster.cc:1428] Setting key b242ec31d5760c7e15b94f79ae4e04c3
I20260927 00:47:15.964583 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:15.956530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43835 (local address 127.0.98.3:47375)
0927 00:47:15.957554 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:15.957612 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:15.959160 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:15.960135 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:15.960146 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:15.960359 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:15.961851 (+ 1492us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:47:15.961853 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:15.961916 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:47:15.962073 (+ 157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:15.962478 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:15.962489 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:15.962496 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:47:15.962710 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 00:47:15.963147 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":173,"threads_started":1}
I20260927 00:47:15.966094 5466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43835
I20260927 00:47:15.966385 5466 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:15.966923 5466 heartbeater.cc:511] Master 127.0.98.62:43835 requested a full tablet report, sending...
I20260927 00:47:15.968356 5016 ts_manager.cc:194] Registered new tserver with Master: bfaa1283912c4692a772ab1c175028fa (127.0.98.3:41979)
I20260927 00:47:15.978582 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:47:15.983042 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:15.981163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37240 (local address 127.0.98.62:43835)
0927 00:47:15.981351 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:15.981361 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:15.981553 (+ 192us) server_negotiation.cc:407] Connection header received
0927 00:47:15.981630 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:15.981633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:15.981677 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:15.981759 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:15.982587 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:15.982590 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:15.982592 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:47:15.982603 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:15.982627 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:47:15.982636 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:15.982722 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:15.982783 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 00:47:15.982822 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 00:47:15.986402 5016 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:47:15.990568 5016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:47:16.005035 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:16.001183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35159 (local address 127.0.0.1:59182)
0927 00:47:16.002040 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:16.002122 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:16.002257 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:16.003027 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:16.003034 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:16.003125 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:16.004064 (+ 939us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:47:16.004066 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:16.004093 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:47:16.004184 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:16.004636 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:16.004647 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:16.004655 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 00:47:16.004759 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 00:47:16.004856 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":98,"threads_started":1}
I20260927 00:47:16.005627 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:16.001231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42989 (local address 127.0.0.1:46894)
0927 00:47:16.002721 (+ 1490us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:16.002732 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:16.002832 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:16.004030 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:16.004034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:16.004052 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:16.004527 (+ 475us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:47:16.004529 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:16.004544 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:47:16.004572 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:16.005366 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:16.005369 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:16.005372 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 00:47:16.005443 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 00:47:16.005478 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1436,"thread_start_us":135,"threads_started":1}
I20260927 00:47:16.005837 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:16.001344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59182 (local address 127.0.98.1:35159)
0927 00:47:16.001885 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:16.001900 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:16.002260 (+ 360us) server_negotiation.cc:407] Connection header received
0927 00:47:16.002379 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:16.002390 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:16.002866 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:16.003031 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:16.004373 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:16.004381 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:16.004397 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:47:16.004414 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:16.004473 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:47:16.004518 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:16.005470 (+ 952us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:16.005524 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 00:47:16.005637 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":210,"threads_started":1}
I20260927 00:47:16.006366 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:16.002207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46894 (local address 127.0.98.2:42989)
0927 00:47:16.002565 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:16.002578 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:16.002833 (+ 255us) server_negotiation.cc:407] Connection header received
0927 00:47:16.002930 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:16.002936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:16.003156 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:16.003304 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:16.005115 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:16.005121 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:16.005130 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:47:16.005140 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:16.005178 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:47:16.005201 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:16.005322 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:16.006065 (+ 743us) server_negotiation.cc:299] Negotiation successful
0927 00:47:16.006151 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":147,"threads_started":1}
I20260927 00:47:16.006922 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:16.002474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41979 (local address 127.0.0.1:52894)
0927 00:47:16.004763 (+ 2289us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:16.004779 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:16.004858 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:16.005521 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:16.005524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:16.005540 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:16.006187 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:47:16.006189 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:16.006200 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:47:16.006226 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:16.006674 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:16.006677 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:16.006680 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 00:47:16.006766 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 00:47:16.006800 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2218,"thread_start_us":103,"threads_started":1}
I20260927 00:47:16.007233 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:16.003531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52894 (local address 127.0.98.3:41979)
0927 00:47:16.004616 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:16.004630 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:16.004886 (+ 256us) server_negotiation.cc:407] Connection header received
0927 00:47:16.004960 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:16.004966 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:16.005188 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:16.005337 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:16.006379 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:16.006388 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:16.006401 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:47:16.006418 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:16.006463 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:47:16.006505 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:16.006628 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:16.006817 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 00:47:16.006897 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":949,"thread_start_us":127,"threads_started":1}
I20260927 00:47:16.009794 5138 tablet_service.cc:1467] Processing CreateTablet for tablet 297129576e3c43d7b320ec3393f3e1a3 (DEFAULT_TABLE table=test-table [id=6fd5cbdaf737424781fa75318e991995]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:16.009799 5269 tablet_service.cc:1467] Processing CreateTablet for tablet 297129576e3c43d7b320ec3393f3e1a3 (DEFAULT_TABLE table=test-table [id=6fd5cbdaf737424781fa75318e991995]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:16.011265 5269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 297129576e3c43d7b320ec3393f3e1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:16.011266 5138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 297129576e3c43d7b320ec3393f3e1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:16.010792 5400 tablet_service.cc:1467] Processing CreateTablet for tablet 297129576e3c43d7b320ec3393f3e1a3 (DEFAULT_TABLE table=test-table [id=6fd5cbdaf737424781fa75318e991995]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:16.012002 5400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 297129576e3c43d7b320ec3393f3e1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:16.019467 5483 tablet_bootstrap.cc:492] T 297129576e3c43d7b320ec3393f3e1a3 P 42c050bcb1e24c5197bceb4457ff2687: Bootstrap starting.
I20260927 00:47:16.020342 5484 tablet_bootstrap.cc:492] T 297129576e3c43d7b320ec3393f3e1a3 P ec7742562bfa4f88a671e738bd983b56: Bootstrap starting.
I20260927 00:47:16.021883 5485 tablet_bootstrap.cc:492] T 297129576e3c43d7b320ec3393f3e1a3 P bfaa1283912c4692a772ab1c175028fa: Bootstrap starting.
I20260927 00:47:16.022889 5483 tablet_bootstrap.cc:654] T 297129576e3c43d7b320ec3393f3e1a3 P 42c050bcb1e24c5197bceb4457ff2687: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:16.022899 5484 tablet_bootstrap.cc:654] T 297129576e3c43d7b320ec3393f3e1a3 P ec7742562bfa4f88a671e738bd983b56: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:16.023811 5483 log.cc:824] T 297129576e3c43d7b320ec3393f3e1a3 P 42c050bcb1e24c5197bceb4457ff2687: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:16.023830 5484 log.cc:824] T 297129576e3c43d7b320ec3393f3e1a3 P ec7742562bfa4f88a671e738bd983b56: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:16.024307 5485 tablet_bootstrap.cc:654] T 297129576e3c43d7b320ec3393f3e1a3 P bfaa1283912c4692a772ab1c175028fa: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:16.025146 5485 log.cc:824] T 297129576e3c43d7b320ec3393f3e1a3 P bfaa1283912c4692a772ab1c175028fa: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:16.026473 5483 tablet_bootstrap.cc:492] T 297129576e3c43d7b320ec3393f3e1a3 P 42c050bcb1e24c5197bceb4457ff2687: No bootstrap required, opened a new log
I20260927 00:47:16.026738 5483 ts_tablet_manager.cc:1397] T 297129576e3c43d7b320ec3393f3e1a3 P 42c050bcb1e24c5197bceb4457ff2687: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260927 00:47:16.026979 5484 tablet_bootstrap.cc:492] T 297129576e3c43d7b320ec3393f3e1a3 P ec7742562bfa4f88a671e738bd983b56: No bootstrap required, opened a new log
I20260927 00:47:16.027288 5484 ts_tablet_manager.cc:1397] T 297129576e3c43d7b320ec3393f3e1a3 P ec7742562bfa4f88a671e738bd983b56: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260927 00:47:16.027323 5485 tablet_bootstrap.cc:492] T 297129576e3c43d7b320ec3393f3e1a3 P bfaa1283912c4692a772ab1c175028fa: No bootstrap required, opened a new log
I20260927 00:47:16.027491 5485 ts_tablet_manager.cc:1397] T 297129576e3c43d7b320ec3393f3e1a3 P bfaa1283912c4692a772ab1c175028fa: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 00:47:16.031440 5484 raft_consensus.cc:348] T 297129576e3c43d7b320ec3393f3e1a3 P ec7742562bfa4f88a671e738bd983b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c050bcb1e24c5197bceb4457ff2687" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35159 } } peers { permanent_uuid: "ec7742562bfa4f88a671e738bd983b56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42989 } } peers { permanent_uuid: "bfaa1283912c4692a772ab1c175028fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41979 } }
I20260927 00:47:16.031821 5484 raft_consensus.cc:374] T 297129576e3c43d7b320ec3393f3e1a3 P ec7742562bfa4f88a671e738bd983b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:16.031880 5484 raft_consensus.cc:729] T 297129576e3c43d7b320ec3393f3e1a3 P ec7742562bfa4f88a671e738bd983b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec7742562bfa4f88a671e738bd983b56, State: Initialized, Role: FOLLOWER
I20260927 00:47:16.032075 5483 raft_consensus.cc:348] T 297129576e3c43d7b320ec3393f3e1a3 P 42c050bcb1e24c5197bceb4457ff2687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c050bcb1e24c5197bceb4457ff2687" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35159 } } peers { permanent_uuid: "ec7742562bfa4f88a671e738bd983b56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42989 } } peers { permanent_uuid: "bfaa1283912c4692a772ab1c175028fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41979 } }
I20260927 00:47:16.032418 5483 raft_consensus.cc:374] T 297129576e3c43d7b320ec3393f3e1a3 P 42c050bcb1e24c5197bceb4457ff2687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:16.032473 5483 raft_consensus.cc:729] T 297129576e3c43d7b320ec3393f3e1a3 P 42c050bcb1e24c5197bceb4457ff2687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42c050bcb1e24c5197bceb4457ff2687, State: Initialized, Role: FOLLOWER
I20260927 00:47:16.032379 5485 raft_consensus.cc:348] T 297129576e3c43d7b320ec3393f3e1a3 P bfaa1283912c4692a772ab1c175028fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c050bcb1e24c5197bceb4457ff2687" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35159 } } peers { permanent_uuid: "ec7742562bfa4f88a671e738bd983b56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42989 } } peers { permanent_uuid: "bfaa1283912c4692a772ab1c175028fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41979 } }
I20260927 00:47:16.032634 5485 raft_consensus.cc:374] T 297129576e3c43d7b320ec3393f3e1a3 P bfaa1283912c4692a772ab1c175028fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:16.032543 5484 consensus_queue.cc:260] T 297129576e3c43d7b320ec3393f3e1a3 P ec7742562bfa4f88a671e738bd983b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c050bcb1e24c5197bceb4457ff2687" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35159 } } peers { permanent_uuid: "ec7742562bfa4f88a671e738bd983b56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42989 } } peers { permanent_uuid: "bfaa1283912c4692a772ab1c175028fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41979 } }
I20260927 00:47:16.032694 5485 raft_consensus.cc:729] T 297129576e3c43d7b320ec3393f3e1a3 P bfaa1283912c4692a772ab1c175028fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfaa1283912c4692a772ab1c175028fa, State: Initialized, Role: FOLLOWER
I20260927 00:47:16.033046 5483 consensus_queue.cc:260] T 297129576e3c43d7b320ec3393f3e1a3 P 42c050bcb1e24c5197bceb4457ff2687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c050bcb1e24c5197bceb4457ff2687" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35159 } } peers { permanent_uuid: "ec7742562bfa4f88a671e738bd983b56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42989 } } peers { permanent_uuid: "bfaa1283912c4692a772ab1c175028fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41979 } }
I20260927 00:47:16.033131 5485 consensus_queue.cc:260] T 297129576e3c43d7b320ec3393f3e1a3 P bfaa1283912c4692a772ab1c175028fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c050bcb1e24c5197bceb4457ff2687" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35159 } } peers { permanent_uuid: "ec7742562bfa4f88a671e738bd983b56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42989 } } peers { permanent_uuid: "bfaa1283912c4692a772ab1c175028fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41979 } }
I20260927 00:47:16.033975 5484 ts_tablet_manager.cc:1428] T 297129576e3c43d7b320ec3393f3e1a3 P ec7742562bfa4f88a671e738bd983b56: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 00:47:16.034173 5483 ts_tablet_manager.cc:1428] T 297129576e3c43d7b320ec3393f3e1a3 P 42c050bcb1e24c5197bceb4457ff2687: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260927 00:47:16.034173 5485 ts_tablet_manager.cc:1428] T 297129576e3c43d7b320ec3393f3e1a3 P bfaa1283912c4692a772ab1c175028fa: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260927 00:47:16.034473 5335 heartbeater.cc:503] Master 127.0.98.62:43835 was elected leader, sending a full tablet report...
I20260927 00:47:16.034498 5204 heartbeater.cc:503] Master 127.0.98.62:43835 was elected leader, sending a full tablet report...
I20260927 00:47:16.035122 5466 heartbeater.cc:503] Master 127.0.98.62:43835 was elected leader, sending a full tablet report...
I20260927 00:47:16.107441 5490 raft_consensus.cc:482] T 297129576e3c43d7b320ec3393f3e1a3 P 42c050bcb1e24c5197bceb4457ff2687 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:47:16.107800 5490 raft_consensus.cc:504] T 297129576e3c43d7b320ec3393f3e1a3 P 42c050bcb1e24c5197bceb4457ff2687 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c050bcb1e24c5197bceb4457ff2687" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35159 } } peers { permanent_uuid: "ec7742562bfa4f88a671e738bd983b56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42989 } } peers { permanent_uuid: "bfaa1283912c4692a772ab1c175028fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41979 } }
I20260927 00:47:16.109817 5490 leader_election.cc:288] T 297129576e3c43d7b320ec3393f3e1a3 P 42c050bcb1e24c5197bceb4457ff2687 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec7742562bfa4f88a671e738bd983b56 (127.0.98.2:42989), bfaa1283912c4692a772ab1c175028fa (127.0.98.3:41979)
I20260927 00:47:16.113111 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:16.110260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34093 (local address 127.0.98.3:41979)
0927 00:47:16.110423 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:16.110432 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:16.110835 (+ 403us) server_negotiation.cc:407] Connection header received
0927 00:47:16.111086 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:16.111090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:16.111153 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:16.111248 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:16.112584 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:16.112588 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:16.112592 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:47:16.112611 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:16.112640 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:47:16.112653 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:16.112781 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:16.112866 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 00:47:16.112922 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:47:16.113153 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:16.110368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37057 (local address 127.0.98.2:42989)
0927 00:47:16.110552 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:16.110557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:16.110953 (+ 396us) server_negotiation.cc:407] Connection header received
0927 00:47:16.111100 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:16.111103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:16.111155 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:16.111248 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:16.112590 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:16.112594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:16.112597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:47:16.112612 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:16.112640 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:47:16.112653 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:16.112781 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:16.112866 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 00:47:16.112922 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 00:47:16.113138 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:16.110111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41979 (local address 127.0.98.1:34093)
0927 00:47:16.110718 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:16.110735 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:16.110930 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:16.111290 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:16.111293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:16.111312 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:16.112311 (+ 999us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:47:16.112313 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:16.112320 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:47:16.112360 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:16.112765 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:16.112769 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:16.112773 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:47:16.112862 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 00:47:16.112927 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":134,"threads_started":1}
I20260927 00:47:16.113138 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:16.110113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42989 (local address 127.0.98.1:37057)
0927 00:47:16.110868 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:16.110881 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:16.111008 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:16.111290 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:16.111293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:16.111312 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:16.112301 (+ 989us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:47:16.112303 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:16.112320 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:47:16.112360 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:16.112765 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:16.112769 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:16.112773 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:47:16.112862 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 00:47:16.112937 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":337,"thread_start_us":164,"threads_started":1}
I20260927 00:47:16.114075 5420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297129576e3c43d7b320ec3393f3e1a3" candidate_uuid: "42c050bcb1e24c5197bceb4457ff2687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaa1283912c4692a772ab1c175028fa" is_pre_election: true
I20260927 00:47:16.114075 5289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297129576e3c43d7b320ec3393f3e1a3" candidate_uuid: "42c050bcb1e24c5197bceb4457ff2687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec7742562bfa4f88a671e738bd983b56" is_pre_election: true
I20260927 00:47:16.114534 5289 raft_consensus.cc:2454] T 297129576e3c43d7b320ec3393f3e1a3 P ec7742562bfa4f88a671e738bd983b56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42c050bcb1e24c5197bceb4457ff2687 in term 0.
I20260927 00:47:16.114534 5420 raft_consensus.cc:2454] T 297129576e3c43d7b320ec3393f3e1a3 P bfaa1283912c4692a772ab1c175028fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42c050bcb1e24c5197bceb4457ff2687 in term 0.
I20260927 00:47:16.115228 5091 leader_election.cc:302] T 297129576e3c43d7b320ec3393f3e1a3 P 42c050bcb1e24c5197bceb4457ff2687 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 42c050bcb1e24c5197bceb4457ff2687, bfaa1283912c4692a772ab1c175028fa, ec7742562bfa4f88a671e738bd983b56; no voters:
I20260927 00:47:16.115573 5490 raft_consensus.cc:2781] T 297129576e3c43d7b320ec3393f3e1a3 P 42c050bcb1e24c5197bceb4457ff2687 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:47:16.115672 5490 raft_consensus.cc:482] T 297129576e3c43d7b320ec3393f3e1a3 P 42c050bcb1e24c5197bceb4457ff2687 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:47:16.115718 5490 raft_consensus.cc:3037] T 297129576e3c43d7b320ec3393f3e1a3 P 42c050bcb1e24c5197bceb4457ff2687 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:16.117172 5490 raft_consensus.cc:504] T 297129576e3c43d7b320ec3393f3e1a3 P 42c050bcb1e24c5197bceb4457ff2687 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c050bcb1e24c5197bceb4457ff2687" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35159 } } peers { permanent_uuid: "ec7742562bfa4f88a671e738bd983b56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42989 } } peers { permanent_uuid: "bfaa1283912c4692a772ab1c175028fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41979 } }
I20260927 00:47:16.117628 5490 leader_election.cc:288] T 297129576e3c43d7b320ec3393f3e1a3 P 42c050bcb1e24c5197bceb4457ff2687 [CANDIDATE]: Term 1 election: Requested vote from peers ec7742562bfa4f88a671e738bd983b56 (127.0.98.2:42989), bfaa1283912c4692a772ab1c175028fa (127.0.98.3:41979)
I20260927 00:47:16.118207 5289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297129576e3c43d7b320ec3393f3e1a3" candidate_uuid: "42c050bcb1e24c5197bceb4457ff2687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec7742562bfa4f88a671e738bd983b56"
I20260927 00:47:16.118207 5420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297129576e3c43d7b320ec3393f3e1a3" candidate_uuid: "42c050bcb1e24c5197bceb4457ff2687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfaa1283912c4692a772ab1c175028fa"
I20260927 00:47:16.118381 5420 raft_consensus.cc:3037] T 297129576e3c43d7b320ec3393f3e1a3 P bfaa1283912c4692a772ab1c175028fa [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:16.118382 5289 raft_consensus.cc:3037] T 297129576e3c43d7b320ec3393f3e1a3 P ec7742562bfa4f88a671e738bd983b56 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:16.120031 5289 raft_consensus.cc:2454] T 297129576e3c43d7b320ec3393f3e1a3 P ec7742562bfa4f88a671e738bd983b56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42c050bcb1e24c5197bceb4457ff2687 in term 1.
I20260927 00:47:16.120039 5420 raft_consensus.cc:2454] T 297129576e3c43d7b320ec3393f3e1a3 P bfaa1283912c4692a772ab1c175028fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42c050bcb1e24c5197bceb4457ff2687 in term 1.
I20260927 00:47:16.120483 5094 leader_election.cc:302] T 297129576e3c43d7b320ec3393f3e1a3 P 42c050bcb1e24c5197bceb4457ff2687 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42c050bcb1e24c5197bceb4457ff2687, bfaa1283912c4692a772ab1c175028fa; no voters:
I20260927 00:47:16.121819 5490 raft_consensus.cc:2781] T 297129576e3c43d7b320ec3393f3e1a3 P 42c050bcb1e24c5197bceb4457ff2687 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:16.122134 5490 raft_consensus.cc:686] T 297129576e3c43d7b320ec3393f3e1a3 P 42c050bcb1e24c5197bceb4457ff2687 [term 1 LEADER]: Becoming Leader. State: Replica: 42c050bcb1e24c5197bceb4457ff2687, State: Running, Role: LEADER
I20260927 00:47:16.122593 5490 consensus_queue.cc:237] T 297129576e3c43d7b320ec3393f3e1a3 P 42c050bcb1e24c5197bceb4457ff2687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c050bcb1e24c5197bceb4457ff2687" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35159 } } peers { permanent_uuid: "ec7742562bfa4f88a671e738bd983b56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42989 } } peers { permanent_uuid: "bfaa1283912c4692a772ab1c175028fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41979 } }
I20260927 00:47:16.127094 5014 catalog_manager.cc:5495] T 297129576e3c43d7b320ec3393f3e1a3 P 42c050bcb1e24c5197bceb4457ff2687 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42c050bcb1e24c5197bceb4457ff2687 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "42c050bcb1e24c5197bceb4457ff2687" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c050bcb1e24c5197bceb4457ff2687" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec7742562bfa4f88a671e738bd983b56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfaa1283912c4692a772ab1c175028fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41979 } health_report { overall_health: UNKNOWN } } }
I20260927 00:47:16.142741 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:16.140965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59192 (local address 127.0.98.1:35159)
0927 00:47:16.141114 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:16.141119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:16.141136 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:47:16.141296 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:16.141299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:16.141381 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:16.141463 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:16.142325 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:16.142329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:16.142331 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:47:16.142345 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:16.142371 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:47:16.142381 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:16.142470 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:16.142541 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 00:47:16.142584 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:47:16.151643 5420 raft_consensus.cc:1261] T 297129576e3c43d7b320ec3393f3e1a3 P bfaa1283912c4692a772ab1c175028fa [term 1 FOLLOWER]: Refusing update from remote peer 42c050bcb1e24c5197bceb4457ff2687: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:47:16.151643 5289 raft_consensus.cc:1261] T 297129576e3c43d7b320ec3393f3e1a3 P ec7742562bfa4f88a671e738bd983b56 [term 1 FOLLOWER]: Refusing update from remote peer 42c050bcb1e24c5197bceb4457ff2687: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:47:16.153214 5490 consensus_queue.cc:1035] T 297129576e3c43d7b320ec3393f3e1a3 P 42c050bcb1e24c5197bceb4457ff2687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfaa1283912c4692a772ab1c175028fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:47:16.153606 5494 consensus_queue.cc:1035] T 297129576e3c43d7b320ec3393f3e1a3 P 42c050bcb1e24c5197bceb4457ff2687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec7742562bfa4f88a671e738bd983b56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:47:16.165354 5497 mvcc.cc:205] Tried to move back new op lower bound from 7333765268064845824 to 7333765267962105856. Current Snapshot: MvccSnapshot[applied={T|T < 7333765268064845824}]
I20260927 00:47:16.166678 5499 mvcc.cc:205] Tried to move back new op lower bound from 7333765268064845824 to 7333765267962105856. Current Snapshot: MvccSnapshot[applied={T|T < 7333765268064845824}]
I20260927 00:47:16.178748 5015 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:47:16.178978 5015 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:47:16.182335 5015 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 41bfecc9d1854a778eb088372d0ae3a7: Sending DeleteTablet for 3 replicas of tablet 297129576e3c43d7b320ec3393f3e1a3
I20260927 00:47:16.183781 5138 tablet_service.cc:1514] Processing DeleteTablet for tablet 297129576e3c43d7b320ec3393f3e1a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:16 UTC) from {username='slave'} at 127.0.0.1:59182
I20260927 00:47:16.183781 5269 tablet_service.cc:1514] Processing DeleteTablet for tablet 297129576e3c43d7b320ec3393f3e1a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:16 UTC) from {username='slave'} at 127.0.0.1:46894
I20260927 00:47:16.183781 5400 tablet_service.cc:1514] Processing DeleteTablet for tablet 297129576e3c43d7b320ec3393f3e1a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:16 UTC) from {username='slave'} at 127.0.0.1:52894
I20260927 00:47:16.184726 5508 tablet_replica.cc:330] T 297129576e3c43d7b320ec3393f3e1a3 P 42c050bcb1e24c5197bceb4457ff2687: stopping tablet replica
I20260927 00:47:16.184938 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 5076
I20260927 00:47:16.186944 5509 tablet_replica.cc:330] T 297129576e3c43d7b320ec3393f3e1a3 P bfaa1283912c4692a772ab1c175028fa: stopping tablet replica
I20260927 00:47:16.187409 5509 raft_consensus.cc:2229] T 297129576e3c43d7b320ec3393f3e1a3 P bfaa1283912c4692a772ab1c175028fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:47:16.187915 5509 raft_consensus.cc:2258] T 297129576e3c43d7b320ec3393f3e1a3 P bfaa1283912c4692a772ab1c175028fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:16.190037 5509 ts_tablet_manager.cc:1905] T 297129576e3c43d7b320ec3393f3e1a3 P bfaa1283912c4692a772ab1c175028fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:16.190421 5507 tablet_replica.cc:330] T 297129576e3c43d7b320ec3393f3e1a3 P ec7742562bfa4f88a671e738bd983b56: stopping tablet replica
I20260927 00:47:16.190796 5507 raft_consensus.cc:2229] T 297129576e3c43d7b320ec3393f3e1a3 P ec7742562bfa4f88a671e738bd983b56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:47:16.191219 5507 raft_consensus.cc:2258] T 297129576e3c43d7b320ec3393f3e1a3 P ec7742562bfa4f88a671e738bd983b56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:16.193025 5507 ts_tablet_manager.cc:1905] T 297129576e3c43d7b320ec3393f3e1a3 P ec7742562bfa4f88a671e738bd983b56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:16.194438 5509 ts_tablet_manager.cc:1918] T 297129576e3c43d7b320ec3393f3e1a3 P bfaa1283912c4692a772ab1c175028fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:47:16.194567 5509 log.cc:1196] T 297129576e3c43d7b320ec3393f3e1a3 P bfaa1283912c4692a772ab1c175028fa: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-2/wal/wals/297129576e3c43d7b320ec3393f3e1a3
I20260927 00:47:16.194943 5509 ts_tablet_manager.cc:1939] T 297129576e3c43d7b320ec3393f3e1a3 P bfaa1283912c4692a772ab1c175028fa: Deleting consensus metadata
I20260927 00:47:16.195674 5507 ts_tablet_manager.cc:1918] T 297129576e3c43d7b320ec3393f3e1a3 P ec7742562bfa4f88a671e738bd983b56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260927 00:47:16.195724 5003 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35159 (error 108)
I20260927 00:47:16.195803 5507 log.cc:1196] T 297129576e3c43d7b320ec3393f3e1a3 P ec7742562bfa4f88a671e738bd983b56: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790470006085796-392-0/minicluster-data/ts-1/wal/wals/297129576e3c43d7b320ec3393f3e1a3
I20260927 00:47:16.196125 5507 ts_tablet_manager.cc:1939] T 297129576e3c43d7b320ec3393f3e1a3 P ec7742562bfa4f88a671e738bd983b56: Deleting consensus metadata
I20260927 00:47:16.196561 5003 catalog_manager.cc:4841] TS bfaa1283912c4692a772ab1c175028fa (127.0.98.3:41979): tablet 297129576e3c43d7b320ec3393f3e1a3 (table test-table [id=6fd5cbdaf737424781fa75318e991995]) successfully deleted
I20260927 00:47:16.197330 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:16.196959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35159 (local address 127.0.0.1:59208)
0927 00:47:16.197110 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:16.197194 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 00:47:16.197468 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 5207
I20260927 00:47:16.197487 5000 catalog_manager.cc:4841] TS ec7742562bfa4f88a671e738bd983b56 (127.0.98.2:42989): tablet 297129576e3c43d7b320ec3393f3e1a3 (table test-table [id=6fd5cbdaf737424781fa75318e991995]) successfully deleted
W20260927 00:47:16.200809 5003 catalog_manager.cc:4584] TS 42c050bcb1e24c5197bceb4457ff2687 (127.0.98.1:35159): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 297129576e3c43d7b320ec3393f3e1a3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35159: connect: Connection refused (error 111)
I20260927 00:47:16.204284 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 5338
I20260927 00:47:16.211130 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 4985
2026-09-27T00:47:16Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1248 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 00:47:16.235028 392 mini_oidc.cc:152] Starting JWKS server
I20260927 00:47:16.237164 392 webserver.cc:460] Webserver started at https://127.0.0.1:45815/ using document root <none> and password file <none>
I20260927 00:47:16.237391 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 00:47:16.237715 392 webserver.cc:460] Webserver started at http://127.0.0.1:33681/ using document root <none> and password file <none>
2026-09-27T00:47:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:47:16Z Disabled control of system clock
I20260927 00:47:16.249346 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43031
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43031
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45815/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470006085796-392-0/testchainca.cert with env {}
W20260927 00:47:16.385896 5516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:16.386276 5516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:16.386322 5516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:16.390683 5516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:47:16.390772 5516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:16.390789 5516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:16.390807 5516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:47:16.390823 5516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 00:47:16.390841 5516 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 00:47:16.390854 5516 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45815/jwks.json
I20260927 00:47:16.397106 5516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43031
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45815/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470006085796-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:16.398550 5516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:16.399866 5516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:16.411131 5521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:16.411172 5524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:16.411214 5522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:16.411461 5516 server_base.cc:1034] running on GCE node
I20260927 00:47:16.412438 5516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:16.414100 5516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:16.415305 5516 hybrid_clock.cc:648] HybridClock initialized: now 1790470036415282 us; error 40 us; skew 500 ppm
I20260927 00:47:16.422092 5516 webserver.cc:460] Webserver started at http://127.0.98.62:37659/ using document root <none> and password file <none>
I20260927 00:47:16.422999 5516 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:16.423123 5516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:16.423415 5516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:16.426254 5516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "52bb847ff1654ac2aeb6c9ff62cb2917"
format_stamp: "Formatted at 2026-09-27 00:47:16 on dist-test-slave-86rs"
server_key: "53eb7e2a3cf7d5c1b8214c12f39d62b9"
server_key_iv: "9922d1934213505ab13b566d20d24ad0"
server_key_version: "encryptionkey@0"
I20260927 00:47:16.426990 5516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "52bb847ff1654ac2aeb6c9ff62cb2917"
format_stamp: "Formatted at 2026-09-27 00:47:16 on dist-test-slave-86rs"
server_key: "53eb7e2a3cf7d5c1b8214c12f39d62b9"
server_key_iv: "9922d1934213505ab13b566d20d24ad0"
server_key_version: "encryptionkey@0"
I20260927 00:47:16.431928 5516 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 00:47:16.435408 5530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:16.437541 5516 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:47:16.437757 5516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "52bb847ff1654ac2aeb6c9ff62cb2917"
format_stamp: "Formatted at 2026-09-27 00:47:16 on dist-test-slave-86rs"
server_key: "53eb7e2a3cf7d5c1b8214c12f39d62b9"
server_key_iv: "9922d1934213505ab13b566d20d24ad0"
server_key_version: "encryptionkey@0"
I20260927 00:47:16.438007 5516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:16.457548 5516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:16.458613 5516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:16.458884 5516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:16.468461 5516 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43031
I20260927 00:47:16.468533 5582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43031 every 8 connection(s)
I20260927 00:47:16.469861 5516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:47:16.474421 5583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:16.477713 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 5516
I20260927 00:47:16.477871 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:47:16.478250 392 external_mini_cluster.cc:1428] Setting key 79c1540016ddffeb920b6638d9b74893
I20260927 00:47:16.483062 5583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52bb847ff1654ac2aeb6c9ff62cb2917: Bootstrap starting.
I20260927 00:47:16.486290 5583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52bb847ff1654ac2aeb6c9ff62cb2917: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:16.487308 5583 log.cc:824] T 00000000000000000000000000000000 P 52bb847ff1654ac2aeb6c9ff62cb2917: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:16.490434 5583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52bb847ff1654ac2aeb6c9ff62cb2917: No bootstrap required, opened a new log
I20260927 00:47:16.491110 5586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:16.480277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33454 (local address 127.0.98.62:43031)
0927 00:47:16.481211 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:16.481232 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:16.481282 (+ 50us) server_negotiation.cc:407] Connection header received
0927 00:47:16.482095 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:16.482120 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:16.482723 (+ 603us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:16.483199 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:16.483801 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:16.485099 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:16.486213 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:16.486894 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:16.487115 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:16.488473 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:16.488479 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:16.488499 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:47:16.489032 (+ 533us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:16.489108 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:47:16.489132 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:16.489230 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:16.489516 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 00:47:16.489950 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":177,"threads_started":1}
I20260927 00:47:16.495529 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:47:16.495986 5583 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52bb847ff1654ac2aeb6c9ff62cb2917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bb847ff1654ac2aeb6c9ff62cb2917" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43031 } }
I20260927 00:47:16.496433 5583 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52bb847ff1654ac2aeb6c9ff62cb2917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:16.496548 5583 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52bb847ff1654ac2aeb6c9ff62cb2917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52bb847ff1654ac2aeb6c9ff62cb2917, State: Initialized, Role: FOLLOWER
I20260927 00:47:16.497288 5583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52bb847ff1654ac2aeb6c9ff62cb2917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bb847ff1654ac2aeb6c9ff62cb2917" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43031 } }
I20260927 00:47:16.497498 5583 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52bb847ff1654ac2aeb6c9ff62cb2917 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:47:16.497578 5583 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52bb847ff1654ac2aeb6c9ff62cb2917 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:47:16.497706 5583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52bb847ff1654ac2aeb6c9ff62cb2917 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:16.499076 5583 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52bb847ff1654ac2aeb6c9ff62cb2917 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bb847ff1654ac2aeb6c9ff62cb2917" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43031 } }
I20260927 00:47:16.499567 5583 leader_election.cc:302] T 00000000000000000000000000000000 P 52bb847ff1654ac2aeb6c9ff62cb2917 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52bb847ff1654ac2aeb6c9ff62cb2917; no voters:
I20260927 00:47:16.500048 5583 leader_election.cc:288] T 00000000000000000000000000000000 P 52bb847ff1654ac2aeb6c9ff62cb2917 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:47:16.500398 5588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52bb847ff1654ac2aeb6c9ff62cb2917 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:16.500705 5588 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52bb847ff1654ac2aeb6c9ff62cb2917 [term 1 LEADER]: Becoming Leader. State: Replica: 52bb847ff1654ac2aeb6c9ff62cb2917, State: Running, Role: LEADER
I20260927 00:47:16.501148 5588 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52bb847ff1654ac2aeb6c9ff62cb2917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bb847ff1654ac2aeb6c9ff62cb2917" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43031 } }
I20260927 00:47:16.501881 5583 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52bb847ff1654ac2aeb6c9ff62cb2917 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:47:16.503602 5590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52bb847ff1654ac2aeb6c9ff62cb2917 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52bb847ff1654ac2aeb6c9ff62cb2917. Latest consensus state: current_term: 1 leader_uuid: "52bb847ff1654ac2aeb6c9ff62cb2917" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bb847ff1654ac2aeb6c9ff62cb2917" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43031 } } }
I20260927 00:47:16.503773 5590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52bb847ff1654ac2aeb6c9ff62cb2917 [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:16.504281 5597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:47:16.504310 5589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52bb847ff1654ac2aeb6c9ff62cb2917 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52bb847ff1654ac2aeb6c9ff62cb2917" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bb847ff1654ac2aeb6c9ff62cb2917" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43031 } } }
I20260927 00:47:16.504423 5589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52bb847ff1654ac2aeb6c9ff62cb2917 [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:16.508925 5597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:47:16.517052 5597 catalog_manager.cc:1347] Generated new cluster ID: 81a1d61320d5493e9d654c55a45a025d
I20260927 00:47:16.517176 5597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:47:16.531176 5597 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:47:16.532502 5597 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:47:16.543845 5597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52bb847ff1654ac2aeb6c9ff62cb2917: Generated new TSK 0
I20260927 00:47:16.544935 5597 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 00:47:16.576893 5613 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 00:47:16.577050 5613 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 00:47:16.577867 5586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:16.557721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33456 (local address 127.0.98.62:43031)
0927 00:47:16.557924 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:16.557937 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:16.558196 (+ 259us) server_negotiation.cc:407] Connection header received
0927 00:47:16.558489 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:16.558494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:16.558574 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:16.558713 (+ 139us) server_negotiation.cc:226] Negotiated authn=JWT
0927 00:47:16.559084 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:16.560109 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:16.561296 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:16.561721 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:16.561846 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:16.562343 (+ 497us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 00:47:16.577327 (+ 14984us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45815/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 00:47:16.577689 (+ 362us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33456: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45815/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 00:47:16.577977 5586 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33456: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45815/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 00:47:16.579455 5612 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:16.557529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43031 (local address 127.0.0.1:33456)
0927 00:47:16.558067 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:16.558085 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:16.558266 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:16.558768 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:16.558771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:16.558781 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0927 00:47:16.558939 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:16.558946 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:16.560340 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:16.560344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:16.561137 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:16.561149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:16.561849 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:16.561879 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:16.561971 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:16.562044 (+ 73us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 00:47:16.577637 (+ 15593us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45815/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 00:47:16.577913 (+ 276us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43031: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45815/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":172,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:43031: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45815/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 00:47:16.580333 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 5516
2026-09-27T00:47:16Z chronyd exiting
I20260927 00:47:16.618727 392 test_util.cc:175] -----------------------------------------------
I20260927 00:47:16.618863 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790470006085796-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (385 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 00:47:16.620198 392 mini_oidc.cc:152] Starting JWKS server
I20260927 00:47:16.621263 392 webserver.cc:460] Webserver started at https://127.0.0.1:44055/ using document root <none> and password file <none>
I20260927 00:47:16.621404 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 00:47:16.621701 392 webserver.cc:460] Webserver started at http://127.0.0.1:38329/ using document root <none> and password file <none>
2026-09-27T00:47:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:47:16Z Disabled control of system clock
I20260927 00:47:16.632555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40419
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40419
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44055/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470006085796-392-0/testchainca.cert with env {}
W20260927 00:47:16.767845 5619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:16.768276 5619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:16.768324 5619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:16.772804 5619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:47:16.772897 5619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:16.772912 5619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:16.772930 5619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:47:16.772948 5619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 00:47:16.772965 5619 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 00:47:16.772979 5619 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44055/jwks.json
I20260927 00:47:16.779017 5619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40419
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44055/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470006085796-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:16.780505 5619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:16.781829 5619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:16.792418 5625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:16.792418 5624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:16.792495 5619 server_base.cc:1034] running on GCE node
W20260927 00:47:16.792418 5627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:16.793324 5619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:16.794747 5619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:16.795923 5619 hybrid_clock.cc:648] HybridClock initialized: now 1790470036795894 us; error 44 us; skew 500 ppm
I20260927 00:47:16.798334 5619 webserver.cc:460] Webserver started at http://127.0.98.62:40949/ using document root <none> and password file <none>
I20260927 00:47:16.799064 5619 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:16.799132 5619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:16.799337 5619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:16.802147 5619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "339c59deeff84dcbb1e7bb0a3ea4ecb8"
format_stamp: "Formatted at 2026-09-27 00:47:16 on dist-test-slave-86rs"
server_key: "d43cf4b29f3c8fe66fafe88d23f704d5"
server_key_iv: "264745d27ab63229678c026bc4a93f9b"
server_key_version: "encryptionkey@0"
I20260927 00:47:16.802919 5619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "339c59deeff84dcbb1e7bb0a3ea4ecb8"
format_stamp: "Formatted at 2026-09-27 00:47:16 on dist-test-slave-86rs"
server_key: "d43cf4b29f3c8fe66fafe88d23f704d5"
server_key_iv: "264745d27ab63229678c026bc4a93f9b"
server_key_version: "encryptionkey@0"
I20260927 00:47:16.807838 5619 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 00:47:16.811146 5633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:16.812903 5619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:47:16.813079 5619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "339c59deeff84dcbb1e7bb0a3ea4ecb8"
format_stamp: "Formatted at 2026-09-27 00:47:16 on dist-test-slave-86rs"
server_key: "d43cf4b29f3c8fe66fafe88d23f704d5"
server_key_iv: "264745d27ab63229678c026bc4a93f9b"
server_key_version: "encryptionkey@0"
I20260927 00:47:16.813230 5619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:16.832553 5619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:16.833552 5619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:16.833787 5619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:16.842572 5619 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40419
I20260927 00:47:16.842574 5685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40419 every 8 connection(s)
I20260927 00:47:16.843840 5619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:47:16.848158 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:16.850723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 5619
I20260927 00:47:16.850901 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:47:16.851320 392 external_mini_cluster.cc:1428] Setting key fe16de98b516a5cc4585c2a709dd2eff
I20260927 00:47:16.855875 5686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 339c59deeff84dcbb1e7bb0a3ea4ecb8: Bootstrap starting.
I20260927 00:47:16.858860 5686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 339c59deeff84dcbb1e7bb0a3ea4ecb8: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:16.859841 5686 log.cc:824] T 00000000000000000000000000000000 P 339c59deeff84dcbb1e7bb0a3ea4ecb8: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:16.862475 5686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 339c59deeff84dcbb1e7bb0a3ea4ecb8: No bootstrap required, opened a new log
I20260927 00:47:16.862586 5689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:16.853239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37244 (local address 127.0.98.62:40419)
0927 00:47:16.854083 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:16.854101 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:16.854147 (+ 46us) server_negotiation.cc:407] Connection header received
0927 00:47:16.854903 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:16.854925 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:16.855448 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:16.855902 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:16.856476 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:16.857619 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:16.858595 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:16.859200 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:16.859393 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:16.860414 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:16.860420 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:16.860437 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:47:16.860946 (+ 509us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:16.861017 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:47:16.861041 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:16.861160 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:16.861322 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 00:47:16.861651 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":174,"threads_started":1}
I20260927 00:47:16.866382 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:47:16.867489 5686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 339c59deeff84dcbb1e7bb0a3ea4ecb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339c59deeff84dcbb1e7bb0a3ea4ecb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40419 } }
I20260927 00:47:16.867865 5686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 339c59deeff84dcbb1e7bb0a3ea4ecb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:16.867940 5686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 339c59deeff84dcbb1e7bb0a3ea4ecb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 339c59deeff84dcbb1e7bb0a3ea4ecb8, State: Initialized, Role: FOLLOWER
I20260927 00:47:16.868558 5686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 339c59deeff84dcbb1e7bb0a3ea4ecb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339c59deeff84dcbb1e7bb0a3ea4ecb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40419 } }
I20260927 00:47:16.868728 5686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 339c59deeff84dcbb1e7bb0a3ea4ecb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:47:16.868779 5686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 339c59deeff84dcbb1e7bb0a3ea4ecb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:47:16.868865 5686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 339c59deeff84dcbb1e7bb0a3ea4ecb8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:16.869951 5686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 339c59deeff84dcbb1e7bb0a3ea4ecb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339c59deeff84dcbb1e7bb0a3ea4ecb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40419 } }
I20260927 00:47:16.870330 5686 leader_election.cc:302] T 00000000000000000000000000000000 P 339c59deeff84dcbb1e7bb0a3ea4ecb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 339c59deeff84dcbb1e7bb0a3ea4ecb8; no voters:
I20260927 00:47:16.870712 5686 leader_election.cc:288] T 00000000000000000000000000000000 P 339c59deeff84dcbb1e7bb0a3ea4ecb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:47:16.870833 5691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 339c59deeff84dcbb1e7bb0a3ea4ecb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:16.871107 5691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 339c59deeff84dcbb1e7bb0a3ea4ecb8 [term 1 LEADER]: Becoming Leader. State: Replica: 339c59deeff84dcbb1e7bb0a3ea4ecb8, State: Running, Role: LEADER
I20260927 00:47:16.871507 5691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 339c59deeff84dcbb1e7bb0a3ea4ecb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339c59deeff84dcbb1e7bb0a3ea4ecb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40419 } }
I20260927 00:47:16.872427 5686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 339c59deeff84dcbb1e7bb0a3ea4ecb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:47:16.873796 5692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 339c59deeff84dcbb1e7bb0a3ea4ecb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "339c59deeff84dcbb1e7bb0a3ea4ecb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339c59deeff84dcbb1e7bb0a3ea4ecb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40419 } } }
I20260927 00:47:16.873764 5693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 339c59deeff84dcbb1e7bb0a3ea4ecb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 339c59deeff84dcbb1e7bb0a3ea4ecb8. Latest consensus state: current_term: 1 leader_uuid: "339c59deeff84dcbb1e7bb0a3ea4ecb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "339c59deeff84dcbb1e7bb0a3ea4ecb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40419 } } }
I20260927 00:47:16.873908 5692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 339c59deeff84dcbb1e7bb0a3ea4ecb8 [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:16.873908 5693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 339c59deeff84dcbb1e7bb0a3ea4ecb8 [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:16.874361 5700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:47:16.878265 5700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:47:16.885553 5700 catalog_manager.cc:1347] Generated new cluster ID: 99d69173880444b6b425a70d19aff62c
I20260927 00:47:16.885682 5700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:47:16.899169 5700 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:47:16.900583 5700 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:47:16.908746 5700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 339c59deeff84dcbb1e7bb0a3ea4ecb8: Generated new TSK 0
I20260927 00:47:16.909796 5700 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 00:47:16.943214 5716 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 00:47:16.943346 5716 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 00:47:16.944197 5689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:16.927283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37254 (local address 127.0.98.62:40419)
0927 00:47:16.927463 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:16.927471 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:16.927658 (+ 187us) server_negotiation.cc:407] Connection header received
0927 00:47:16.927902 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:16.927905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:16.927976 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:16.928104 (+ 128us) server_negotiation.cc:226] Negotiated authn=JWT
0927 00:47:16.928383 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:16.929027 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:16.930346 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:16.930593 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:16.930686 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:16.931054 (+ 368us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 00:47:16.943760 (+ 12706us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44055/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 00:47:16.944057 (+ 297us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37254: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44055/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 00:47:16.944347 5689 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37254: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44055/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 00:47:16.944547 5715 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:16.927163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40419 (local address 127.0.0.1:37254)
0927 00:47:16.927534 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:16.927552 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:16.927705 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:16.928111 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:16.928115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:16.928123 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 00:47:16.928283 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:16.928291 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:16.929141 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:16.929145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:16.930200 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:16.930210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:16.930695 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:16.930724 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:16.930816 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:16.930854 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 00:47:16.944040 (+ 13186us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44055/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 00:47:16.944324 (+ 284us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40419: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44055/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":136,"threads_started":1}
I20260927 00:47:16.945328 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 5619
2026-09-27T00:47:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (366 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 00:47:16.986572 392 mini_oidc.cc:152] Starting JWKS server
I20260927 00:47:16.987630 392 webserver.cc:460] Webserver started at https://127.0.0.1:33845/ using document root <none> and password file <none>
I20260927 00:47:16.987778 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 00:47:16.988065 392 webserver.cc:460] Webserver started at http://127.0.0.1:41733/ using document root <none> and password file <none>
2026-09-27T00:47:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:47:16Z Disabled control of system clock
I20260927 00:47:16.999259 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42881
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42881
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33845/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 00:47:17.139773 5722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:17.140188 5722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:17.140337 5722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:17.144816 5722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:47:17.144913 5722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:17.144928 5722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:17.144948 5722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:47:17.144965 5722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 00:47:17.144982 5722 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 00:47:17.144996 5722 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33845/jwks.json
I20260927 00:47:17.150538 5722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42881
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33845/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:17.152185 5722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:17.154214 5722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:17.165724 5727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:17.165724 5730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:17.166064 5722 server_base.cc:1034] running on GCE node
W20260927 00:47:17.165724 5728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:17.167002 5722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:17.168721 5722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:17.169924 5722 hybrid_clock.cc:648] HybridClock initialized: now 1790470037169891 us; error 100 us; skew 500 ppm
I20260927 00:47:17.172600 5722 webserver.cc:460] Webserver started at http://127.0.98.62:36617/ using document root <none> and password file <none>
I20260927 00:47:17.173354 5722 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:17.173437 5722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:17.173669 5722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:17.176578 5722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "086efe20a9d44a6ab2811b487005ee84"
format_stamp: "Formatted at 2026-09-27 00:47:17 on dist-test-slave-86rs"
server_key: "9642771f8c26c40877d010df7e3d5ff6"
server_key_iv: "84bdfd6a4df7e557a43961b98b518b2c"
server_key_version: "encryptionkey@0"
I20260927 00:47:17.177354 5722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "086efe20a9d44a6ab2811b487005ee84"
format_stamp: "Formatted at 2026-09-27 00:47:17 on dist-test-slave-86rs"
server_key: "9642771f8c26c40877d010df7e3d5ff6"
server_key_iv: "84bdfd6a4df7e557a43961b98b518b2c"
server_key_version: "encryptionkey@0"
I20260927 00:47:17.182482 5722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 00:47:17.186654 5736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:17.188954 5722 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:47:17.189147 5722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "086efe20a9d44a6ab2811b487005ee84"
format_stamp: "Formatted at 2026-09-27 00:47:17 on dist-test-slave-86rs"
server_key: "9642771f8c26c40877d010df7e3d5ff6"
server_key_iv: "84bdfd6a4df7e557a43961b98b518b2c"
server_key_version: "encryptionkey@0"
I20260927 00:47:17.189313 5722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:17.205963 5722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:17.206954 5722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:17.207197 5722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:17.216874 5788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42881 every 8 connection(s)
I20260927 00:47:17.216874 5722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42881
I20260927 00:47:17.218566 5722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:47:17.223203 5789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:17.227857 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 5722
I20260927 00:47:17.227977 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:47:17.228391 392 external_mini_cluster.cc:1428] Setting key bc685d35a60cee225dfa3af5541775dc
I20260927 00:47:17.231060 5789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 086efe20a9d44a6ab2811b487005ee84: Bootstrap starting.
I20260927 00:47:17.234130 5789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 086efe20a9d44a6ab2811b487005ee84: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:17.235144 5789 log.cc:824] T 00000000000000000000000000000000 P 086efe20a9d44a6ab2811b487005ee84: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:17.238003 5789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 086efe20a9d44a6ab2811b487005ee84: No bootstrap required, opened a new log
I20260927 00:47:17.240303 5792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:17.230185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44662 (local address 127.0.98.62:42881)
0927 00:47:17.231191 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:17.231216 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:17.231294 (+ 78us) server_negotiation.cc:407] Connection header received
0927 00:47:17.232097 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:17.232128 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:17.232744 (+ 616us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:17.233177 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:17.233721 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.234887 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.235939 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.236594 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.236797 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:17.237901 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:17.237907 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:17.237927 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:47:17.238531 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:17.238610 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:47:17.238634 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:17.238748 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:17.238929 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 00:47:17.239272 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":252,"threads_started":1}
I20260927 00:47:17.243050 5789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 086efe20a9d44a6ab2811b487005ee84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086efe20a9d44a6ab2811b487005ee84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42881 } }
I20260927 00:47:17.243445 5789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 086efe20a9d44a6ab2811b487005ee84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:17.243534 5789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 086efe20a9d44a6ab2811b487005ee84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 086efe20a9d44a6ab2811b487005ee84, State: Initialized, Role: FOLLOWER
I20260927 00:47:17.244210 5789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 086efe20a9d44a6ab2811b487005ee84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086efe20a9d44a6ab2811b487005ee84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42881 } }
I20260927 00:47:17.244377 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:47:17.244463 5789 raft_consensus.cc:388] T 00000000000000000000000000000000 P 086efe20a9d44a6ab2811b487005ee84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:47:17.244568 5789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 086efe20a9d44a6ab2811b487005ee84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:47:17.244690 5789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 086efe20a9d44a6ab2811b487005ee84 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:17.246186 5789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 086efe20a9d44a6ab2811b487005ee84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086efe20a9d44a6ab2811b487005ee84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42881 } }
I20260927 00:47:17.246662 5789 leader_election.cc:302] T 00000000000000000000000000000000 P 086efe20a9d44a6ab2811b487005ee84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 086efe20a9d44a6ab2811b487005ee84; no voters:
I20260927 00:47:17.247090 5789 leader_election.cc:288] T 00000000000000000000000000000000 P 086efe20a9d44a6ab2811b487005ee84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:47:17.247244 5794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 086efe20a9d44a6ab2811b487005ee84 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:17.247529 5794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 086efe20a9d44a6ab2811b487005ee84 [term 1 LEADER]: Becoming Leader. State: Replica: 086efe20a9d44a6ab2811b487005ee84, State: Running, Role: LEADER
I20260927 00:47:17.247987 5794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 086efe20a9d44a6ab2811b487005ee84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086efe20a9d44a6ab2811b487005ee84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42881 } }
I20260927 00:47:17.248916 5789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 086efe20a9d44a6ab2811b487005ee84 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:47:17.250056 5795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 086efe20a9d44a6ab2811b487005ee84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "086efe20a9d44a6ab2811b487005ee84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086efe20a9d44a6ab2811b487005ee84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42881 } } }
I20260927 00:47:17.250362 5796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 086efe20a9d44a6ab2811b487005ee84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 086efe20a9d44a6ab2811b487005ee84. Latest consensus state: current_term: 1 leader_uuid: "086efe20a9d44a6ab2811b487005ee84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086efe20a9d44a6ab2811b487005ee84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42881 } } }
I20260927 00:47:17.250522 5796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 086efe20a9d44a6ab2811b487005ee84 [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:17.250483 5795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 086efe20a9d44a6ab2811b487005ee84 [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:17.250955 5803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:47:17.255177 5803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:47:17.262844 5803 catalog_manager.cc:1347] Generated new cluster ID: f1dea3376acf4cdf85edb671b206aead
I20260927 00:47:17.262961 5803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:47:17.281692 5803 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:47:17.282961 5803 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:47:17.289312 5803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 086efe20a9d44a6ab2811b487005ee84: Generated new TSK 0
I20260927 00:47:17.290306 5803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:47:17.315166 5792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:17.308485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44668 (local address 127.0.98.62:42881)
0927 00:47:17.308660 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:17.308667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:17.308830 (+ 163us) server_negotiation.cc:407] Connection header received
0927 00:47:17.309026 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:17.309031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:17.309105 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:17.309260 (+ 155us) server_negotiation.cc:226] Negotiated authn=JWT
0927 00:47:17.309573 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.310439 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.311436 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.311822 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.311947 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:17.312358 (+ 411us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 00:47:17.314559 (+ 2201us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33845/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 00:47:17.314927 (+ 368us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44668: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33845/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 00:47:17.315271 5792 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44668: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33845/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 00:47:17.316937 5818 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:17.308410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42881 (local address 127.0.0.1:44668)
0927 00:47:17.308717 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:17.308735 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:17.308858 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:17.309283 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:17.309286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:17.309294 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 00:47:17.309436 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:17.309442 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.310595 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.310599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:17.311304 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:17.311313 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.311948 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.311977 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:17.312060 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:17.312090 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 00:47:17.314908 (+ 2818us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33845/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 00:47:17.316722 (+ 1814us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42881: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33845/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":1452,"thread_start_us":125,"threads_started":1}
I20260927 00:47:17.317628 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 5722
E20260927 00:47:17.326757 5819 webserver.cc:533] Webserver: sslize failed:
E20260927 00:47:17.326869 5819 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T00:47:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (371 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T00:47:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:47:17Z Disabled control of system clock
I20260927 00:47:17.368523 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43871
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43871
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470006085796-392-0/testchainca.cert with env {}
W20260927 00:47:17.504381 5823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:17.504859 5823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:17.504916 5823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:17.509364 5823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:47:17.509461 5823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:17.509478 5823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:17.509497 5823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:47:17.509514 5823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 00:47:17.509532 5823 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 00:47:17.509546 5823 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 00:47:17.516852 5823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43871
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470006085796-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:17.518505 5823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:17.519827 5823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:17.530305 5831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:17.530308 5829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:17.530390 5823 server_base.cc:1034] running on GCE node
W20260927 00:47:17.530308 5828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:17.531312 5823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:17.532884 5823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:17.534123 5823 hybrid_clock.cc:648] HybridClock initialized: now 1790470037534070 us; error 55 us; skew 500 ppm
I20260927 00:47:17.536648 5823 webserver.cc:460] Webserver started at http://127.0.98.62:38901/ using document root <none> and password file <none>
I20260927 00:47:17.537413 5823 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:17.537518 5823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:17.537796 5823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:17.540452 5823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "e1b59753ada84d379ce1321ab21ba90b"
format_stamp: "Formatted at 2026-09-27 00:47:17 on dist-test-slave-86rs"
server_key: "bf665fb0612714b0747942c75edaabf5"
server_key_iv: "4a574f97bbef65e73a62e326a5271c54"
server_key_version: "encryptionkey@0"
I20260927 00:47:17.541138 5823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "e1b59753ada84d379ce1321ab21ba90b"
format_stamp: "Formatted at 2026-09-27 00:47:17 on dist-test-slave-86rs"
server_key: "bf665fb0612714b0747942c75edaabf5"
server_key_iv: "4a574f97bbef65e73a62e326a5271c54"
server_key_version: "encryptionkey@0"
I20260927 00:47:17.545858 5823 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:47:17.549074 5837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:17.550740 5823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:47:17.550906 5823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "e1b59753ada84d379ce1321ab21ba90b"
format_stamp: "Formatted at 2026-09-27 00:47:17 on dist-test-slave-86rs"
server_key: "bf665fb0612714b0747942c75edaabf5"
server_key_iv: "4a574f97bbef65e73a62e326a5271c54"
server_key_version: "encryptionkey@0"
I20260927 00:47:17.551054 5823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:17.569654 5823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:17.570681 5823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:17.570926 5823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:17.579545 5823 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43871
I20260927 00:47:17.579564 5889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43871 every 8 connection(s)
I20260927 00:47:17.580816 5823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:47:17.585237 5890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:17.586395 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 5823
I20260927 00:47:17.586557 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:47:17.586887 392 external_mini_cluster.cc:1428] Setting key 954c759a4b0d3e9a5e5368ed74f081df
I20260927 00:47:17.593098 5890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1b59753ada84d379ce1321ab21ba90b: Bootstrap starting.
I20260927 00:47:17.596100 5890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1b59753ada84d379ce1321ab21ba90b: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:17.597111 5890 log.cc:824] T 00000000000000000000000000000000 P e1b59753ada84d379ce1321ab21ba90b: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:17.598353 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:17.588730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48170 (local address 127.0.98.62:43871)
0927 00:47:17.589599 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:17.589622 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:17.589670 (+ 48us) server_negotiation.cc:407] Connection header received
0927 00:47:17.590410 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:17.590432 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:17.590964 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:17.591404 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:17.591957 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.593149 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.594157 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.594716 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.594902 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:17.595910 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:17.595916 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:17.595932 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:47:17.596506 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:17.596576 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:47:17.596599 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:17.596728 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:17.596855 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 00:47:17.597186 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":182,"threads_started":1}
I20260927 00:47:17.599993 5890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1b59753ada84d379ce1321ab21ba90b: No bootstrap required, opened a new log
I20260927 00:47:17.602360 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:47:17.605053 5890 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1b59753ada84d379ce1321ab21ba90b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b59753ada84d379ce1321ab21ba90b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43871 } }
I20260927 00:47:17.605393 5890 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1b59753ada84d379ce1321ab21ba90b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:17.605485 5890 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1b59753ada84d379ce1321ab21ba90b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1b59753ada84d379ce1321ab21ba90b, State: Initialized, Role: FOLLOWER
I20260927 00:47:17.606051 5890 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1b59753ada84d379ce1321ab21ba90b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b59753ada84d379ce1321ab21ba90b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43871 } }
I20260927 00:47:17.606225 5890 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1b59753ada84d379ce1321ab21ba90b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:47:17.606319 5890 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1b59753ada84d379ce1321ab21ba90b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:47:17.606462 5890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1b59753ada84d379ce1321ab21ba90b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:17.607687 5890 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1b59753ada84d379ce1321ab21ba90b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b59753ada84d379ce1321ab21ba90b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43871 } }
I20260927 00:47:17.608150 5890 leader_election.cc:302] T 00000000000000000000000000000000 P e1b59753ada84d379ce1321ab21ba90b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1b59753ada84d379ce1321ab21ba90b; no voters:
I20260927 00:47:17.608652 5890 leader_election.cc:288] T 00000000000000000000000000000000 P e1b59753ada84d379ce1321ab21ba90b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:47:17.608742 5895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1b59753ada84d379ce1321ab21ba90b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:17.608969 5895 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1b59753ada84d379ce1321ab21ba90b [term 1 LEADER]: Becoming Leader. State: Replica: e1b59753ada84d379ce1321ab21ba90b, State: Running, Role: LEADER
I20260927 00:47:17.609333 5895 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1b59753ada84d379ce1321ab21ba90b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b59753ada84d379ce1321ab21ba90b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43871 } }
I20260927 00:47:17.610370 5890 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1b59753ada84d379ce1321ab21ba90b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:47:17.611513 5897 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1b59753ada84d379ce1321ab21ba90b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1b59753ada84d379ce1321ab21ba90b. Latest consensus state: current_term: 1 leader_uuid: "e1b59753ada84d379ce1321ab21ba90b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b59753ada84d379ce1321ab21ba90b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43871 } } }
I20260927 00:47:17.611682 5897 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1b59753ada84d379ce1321ab21ba90b [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:17.612169 5904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:47:17.612022 5896 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1b59753ada84d379ce1321ab21ba90b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1b59753ada84d379ce1321ab21ba90b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b59753ada84d379ce1321ab21ba90b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43871 } } }
I20260927 00:47:17.612507 5896 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1b59753ada84d379ce1321ab21ba90b [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:17.616655 5904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:47:17.624436 5904 catalog_manager.cc:1347] Generated new cluster ID: 084d437db5454a3ebfc7da9c97eeb060
I20260927 00:47:17.624575 5904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:47:17.634051 5904 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:47:17.635505 5904 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:47:17.641381 5904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1b59753ada84d379ce1321ab21ba90b: Generated new TSK 0
I20260927 00:47:17.642432 5904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:47:17.726727 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:17.654575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48184 (local address 127.0.98.62:43871)
0927 00:47:17.654910 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:17.654916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:17.655060 (+ 144us) server_negotiation.cc:407] Connection header received
0927 00:47:17.655264 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:17.655268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:17.655355 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:17.655518 (+ 163us) server_negotiation.cc:226] Negotiated authn=JWT
0927 00:47:17.656043 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.657054 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.658269 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.658653 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.658786 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:17.659248 (+ 462us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 00:47:17.726090 (+ 66842us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 00:47:17.726504 (+ 414us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48184: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":177}
W20260927 00:47:17.726845 5893 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48184: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 00:47:17.726900 5919 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:17.654488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43871 (local address 127.0.0.1:48184)
0927 00:47:17.654892 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:17.654918 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:17.655057 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:17.655593 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:17.655597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:17.655606 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 00:47:17.655738 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:17.655744 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.657249 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.657253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:17.658099 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:17.658115 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:17.658784 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:17.658822 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:17.658920 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:17.658958 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 00:47:17.726399 (+ 67441us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 00:47:17.726659 (+ 260us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43871: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":176,"threads_started":1}
I20260927 00:47:17.727761 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 5823
2026-09-27T00:47:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (390 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (151 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (155 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:47:18 dist-test-slave-86rs krb5kdc[5931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:47:18 dist-test-slave-86rs krb5kdc[5931](info): set up 2 sockets
Sep 27 00:47:18 dist-test-slave-86rs krb5kdc[5931](info): commencing operation
krb5kdc: starting...
W20260927 00:47:19.089061 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:47:19 dist-test-slave-86rs krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:47:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:47:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:47:19.226881 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34549
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:19.365646 5944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:19.366096 5944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:19.366181 5944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:19.370939 5944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:47:19.371068 5944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:19.371111 5944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:19.371150 5944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:47:19.371186 5944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:47:19.377560 5944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:19.379153 5944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:19.380601 5944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:19.391103 5949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:19.391091 5952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:19.391091 5950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:19.391194 5944 server_base.cc:1034] running on GCE node
I20260927 00:47:19.392094 5944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:19.393648 5944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:19.394824 5944 hybrid_clock.cc:648] HybridClock initialized: now 1790470039394795 us; error 50 us; skew 500 ppm
Sep 27 00:47:19 dist-test-slave-86rs krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:19.398294 5944 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:47:19.399528 5944 webserver.cc:460] Webserver started at http://127.0.98.62:41945/ using document root <none> and password file <none>
I20260927 00:47:19.400339 5944 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:19.400444 5944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:19.400738 5944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:19.403582 5944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "8dbd325b289b43bf8cfc362f56eb1281"
format_stamp: "Formatted at 2026-09-27 00:47:19 on dist-test-slave-86rs"
server_key: "bdabe57da06f1e1a9e83e2659777a6f8"
server_key_iv: "acae18b857bef77c7e981d5e2394525f"
server_key_version: "encryptionkey@0"
I20260927 00:47:19.404484 5944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "8dbd325b289b43bf8cfc362f56eb1281"
format_stamp: "Formatted at 2026-09-27 00:47:19 on dist-test-slave-86rs"
server_key: "bdabe57da06f1e1a9e83e2659777a6f8"
server_key_iv: "acae18b857bef77c7e981d5e2394525f"
server_key_version: "encryptionkey@0"
I20260927 00:47:19.409505 5944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 00:47:19.412712 5959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:19.414372 5944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:47:19.414556 5944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "8dbd325b289b43bf8cfc362f56eb1281"
format_stamp: "Formatted at 2026-09-27 00:47:19 on dist-test-slave-86rs"
server_key: "bdabe57da06f1e1a9e83e2659777a6f8"
server_key_iv: "acae18b857bef77c7e981d5e2394525f"
server_key_version: "encryptionkey@0"
I20260927 00:47:19.414706 5944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:19.435393 5944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:19.439808 5944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:19.440068 5944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:19.449044 5944 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34549
I20260927 00:47:19.449384 6011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34549 every 8 connection(s)
I20260927 00:47:19.450423 5944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:47:19.454727 6012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:19.455899 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 5944
I20260927 00:47:19.456063 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:47:19.456489 392 external_mini_cluster.cc:1428] Setting key 9781cf578a453430b4a9c84fbd5d8cd2
I20260927 00:47:19.463449 6012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dbd325b289b43bf8cfc362f56eb1281: Bootstrap starting.
I20260927 00:47:19.467185 6012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8dbd325b289b43bf8cfc362f56eb1281: Neither blocks nor log segments found. Creating new log.
Sep 27 00:47:19 dist-test-slave-86rs krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:19.468205 6012 log.cc:824] T 00000000000000000000000000000000 P 8dbd325b289b43bf8cfc362f56eb1281: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:19.470903 6012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dbd325b289b43bf8cfc362f56eb1281: No bootstrap required, opened a new log
I20260927 00:47:19.474220 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:19.458456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41670 (local address 127.0.98.62:34549)
0927 00:47:19.459075 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:19.459097 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:19.459145 (+ 48us) server_negotiation.cc:407] Connection header received
0927 00:47:19.459963 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:19.459994 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:19.460556 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:19.460958 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:19.462003 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:19.463179 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:19.464419 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:19.464958 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:19.465088 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:19.468451 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:19.468458 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:19.468476 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:19.468509 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:19.470483 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:19.471033 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:19.471042 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:19.471050 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:19.471190 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:19.471498 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:19.471501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:19.471503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:19.471982 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:19.472228 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:19.472539 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 00:47:19.472930 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":221,"threads_started":1}
I20260927 00:47:19.475981 6012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8dbd325b289b43bf8cfc362f56eb1281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dbd325b289b43bf8cfc362f56eb1281" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } }
I20260927 00:47:19.476359 6012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8dbd325b289b43bf8cfc362f56eb1281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:19.476444 6012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8dbd325b289b43bf8cfc362f56eb1281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dbd325b289b43bf8cfc362f56eb1281, State: Initialized, Role: FOLLOWER
I20260927 00:47:19.477044 6012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8dbd325b289b43bf8cfc362f56eb1281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dbd325b289b43bf8cfc362f56eb1281" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } }
I20260927 00:47:19.477229 6012 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8dbd325b289b43bf8cfc362f56eb1281 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:47:19.477305 6012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8dbd325b289b43bf8cfc362f56eb1281 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:47:19.477394 6012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8dbd325b289b43bf8cfc362f56eb1281 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:19.478540 6012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8dbd325b289b43bf8cfc362f56eb1281 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dbd325b289b43bf8cfc362f56eb1281" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } }
I20260927 00:47:19.478935 6012 leader_election.cc:302] T 00000000000000000000000000000000 P 8dbd325b289b43bf8cfc362f56eb1281 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8dbd325b289b43bf8cfc362f56eb1281; no voters:
I20260927 00:47:19.479354 6012 leader_election.cc:288] T 00000000000000000000000000000000 P 8dbd325b289b43bf8cfc362f56eb1281 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:47:19.479467 6017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8dbd325b289b43bf8cfc362f56eb1281 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:19.479678 6017 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8dbd325b289b43bf8cfc362f56eb1281 [term 1 LEADER]: Becoming Leader. State: Replica: 8dbd325b289b43bf8cfc362f56eb1281, State: Running, Role: LEADER
I20260927 00:47:19.480063 6017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8dbd325b289b43bf8cfc362f56eb1281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dbd325b289b43bf8cfc362f56eb1281" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } }
I20260927 00:47:19.481087 6012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8dbd325b289b43bf8cfc362f56eb1281 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:47:19.482347 6019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dbd325b289b43bf8cfc362f56eb1281 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dbd325b289b43bf8cfc362f56eb1281. Latest consensus state: current_term: 1 leader_uuid: "8dbd325b289b43bf8cfc362f56eb1281" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dbd325b289b43bf8cfc362f56eb1281" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } } }
I20260927 00:47:19.482366 6018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dbd325b289b43bf8cfc362f56eb1281 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8dbd325b289b43bf8cfc362f56eb1281" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dbd325b289b43bf8cfc362f56eb1281" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34549 } } }
I20260927 00:47:19.482525 6018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dbd325b289b43bf8cfc362f56eb1281 [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:19.482970 6026 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:47:19.482524 6019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dbd325b289b43bf8cfc362f56eb1281 [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:19.486866 6026 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:47:19.493649 6026 catalog_manager.cc:1347] Generated new cluster ID: 7304c2f37e5a4cb1a0af508f8fcdb116
I20260927 00:47:19.493731 6026 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:47:19.521194 6026 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:47:19.522531 6026 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:47:19.531891 6026 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8dbd325b289b43bf8cfc362f56eb1281: Generated new TSK 0
I20260927 00:47:19.533005 6026 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:47:19.563896 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34549
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:19.705821 6038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:19.706207 6038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:19.706259 6038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:19.710543 6038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:19.710628 6038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:19.710700 6038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:47:19.716594 6038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:19.718014 6038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:19.719496 6038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:19.731706 6043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:19.731706 6046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:19.731706 6044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:19.732192 6038 server_base.cc:1034] running on GCE node
I20260927 00:47:19.732810 6038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:19.733748 6038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:19.734962 6038 hybrid_clock.cc:648] HybridClock initialized: now 1790470039734946 us; error 40 us; skew 500 ppm
Sep 27 00:47:19 dist-test-slave-86rs krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:19.738422 6038 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:47:19.739513 6038 webserver.cc:460] Webserver started at http://127.0.98.1:37253/ using document root <none> and password file <none>
I20260927 00:47:19.740192 6038 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:19.740319 6038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:19.740542 6038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:19.743310 6038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-0/data/instance:
uuid: "5148f82ffb574a07b89b1d789b9a5358"
format_stamp: "Formatted at 2026-09-27 00:47:19 on dist-test-slave-86rs"
server_key: "0d855397969c2b9e375eab28204163b2"
server_key_iv: "700995ca3f70421e97a6a291e22cc0d4"
server_key_version: "encryptionkey@0"
I20260927 00:47:19.744114 6038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5148f82ffb574a07b89b1d789b9a5358"
format_stamp: "Formatted at 2026-09-27 00:47:19 on dist-test-slave-86rs"
server_key: "0d855397969c2b9e375eab28204163b2"
server_key_iv: "700995ca3f70421e97a6a291e22cc0d4"
server_key_version: "encryptionkey@0"
I20260927 00:47:19.749624 6038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 00:47:19.752861 6053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:19.754492 6038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:47:19.754673 6038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "5148f82ffb574a07b89b1d789b9a5358"
format_stamp: "Formatted at 2026-09-27 00:47:19 on dist-test-slave-86rs"
server_key: "0d855397969c2b9e375eab28204163b2"
server_key_iv: "700995ca3f70421e97a6a291e22cc0d4"
server_key_version: "encryptionkey@0"
I20260927 00:47:19.754834 6038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:19.767506 6038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:19.771834 6038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:19.772119 6038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:19.773001 6038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:19.774075 6038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:19.774150 6038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:19.774235 6038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:19.774287 6038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:19.786039 6038 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36447
I20260927 00:47:19.786087 6166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36447 every 8 connection(s)
I20260927 00:47:19.787285 6038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:47:19.792929 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 6038
I20260927 00:47:19.793103 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:47:19.793426 392 external_mini_cluster.cc:1428] Setting key 27af79bdbcb601b41d7481020a6b4998
Sep 27 00:47:19 dist-test-slave-86rs krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:47:19.806706 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:19.789809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46269 (local address 127.0.98.62:34549)
0927 00:47:19.790000 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:19.790006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:19.791223 (+ 1217us) server_negotiation.cc:407] Connection header received
0927 00:47:19.793000 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:19.793005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:19.793079 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:19.793177 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:19.794982 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:19.795663 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:19.797812 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:19.798278 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:19.798417 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:19.801904 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:19.801909 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:19.801912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:19.801952 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:19.803632 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:19.804352 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:19.804355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:19.804357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:19.804450 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:19.804883 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:19.804887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:19.804889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:19.805174 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:19.805295 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:19.806117 (+ 822us) server_negotiation.cc:299] Negotiation successful
0927 00:47:19.806299 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 00:47:19.808144 6169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:19.790374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.1:46269)
0927 00:47:19.791005 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:19.791063 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:19.792710 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:19.793408 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:19.793417 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:19.793932 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:19.794724 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:19.794742 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:19.795838 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:19.795845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:19.797660 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:19.797673 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:19.798498 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:19.798502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:19.798702 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:19.799466 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:19.799494 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:19.801680 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:19.803791 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:19.803799 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:19.803825 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:19.804171 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:19.804553 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:19.804557 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:19.804559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:19.804770 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:19.805296 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:19.805308 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:19.805695 (+ 387us) client_negotiation.cc:769] Sending connection context
0927 00:47:19.806004 (+ 309us) client_negotiation.cc:240] Negotiation successful
0927 00:47:19.806523 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":149,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:47:19.810139 6167 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34549
I20260927 00:47:19.810556 6167 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:19.811292 6167 heartbeater.cc:511] Master 127.0.98.62:34549 requested a full tablet report, sending...
I20260927 00:47:19.813592 5976 ts_manager.cc:194] Registered new tserver with Master: 5148f82ffb574a07b89b1d789b9a5358 (127.0.98.1:36447)
I20260927 00:47:19.815112 5976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46269
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:47:19.828833 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34549
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:19.974900 6172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:19.975273 6172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:19.975323 6172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:19.979856 6172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:19.979959 6172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:19.980027 6172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:47:19.986464 6172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:19.988309 6172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:19.989779 6172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:20.002923 6180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:20.002976 6177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:20.003068 6172 server_base.cc:1034] running on GCE node
W20260927 00:47:20.002979 6178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:20.003978 6172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:20.005061 6172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:20.006312 6172 hybrid_clock.cc:648] HybridClock initialized: now 1790470040006283 us; error 52 us; skew 500 ppm
Sep 27 00:47:20 dist-test-slave-86rs krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:20.010602 6172 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:47:20.011989 6172 webserver.cc:460] Webserver started at http://127.0.98.2:41787/ using document root <none> and password file <none>
I20260927 00:47:20.012928 6172 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:20.013058 6172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:20.013382 6172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:20.016470 6172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-1/data/instance:
uuid: "11ce427695804d9f8fb5e61296c8c27e"
format_stamp: "Formatted at 2026-09-27 00:47:20 on dist-test-slave-86rs"
server_key: "0ea789753ef8e036d5a6fbcd7dbfe879"
server_key_iv: "588400eaa47b61e37550bc00612d718e"
server_key_version: "encryptionkey@0"
I20260927 00:47:20.017345 6172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-1/wal/instance:
uuid: "11ce427695804d9f8fb5e61296c8c27e"
format_stamp: "Formatted at 2026-09-27 00:47:20 on dist-test-slave-86rs"
server_key: "0ea789753ef8e036d5a6fbcd7dbfe879"
server_key_iv: "588400eaa47b61e37550bc00612d718e"
server_key_version: "encryptionkey@0"
I20260927 00:47:20.023340 6172 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 00:47:20.027267 6187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:20.029297 6172 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:47:20.029477 6172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "11ce427695804d9f8fb5e61296c8c27e"
format_stamp: "Formatted at 2026-09-27 00:47:20 on dist-test-slave-86rs"
server_key: "0ea789753ef8e036d5a6fbcd7dbfe879"
server_key_iv: "588400eaa47b61e37550bc00612d718e"
server_key_version: "encryptionkey@0"
I20260927 00:47:20.029639 6172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:20.047339 6172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:20.051374 6172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:20.051623 6172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:20.052460 6172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:20.053514 6172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:20.053591 6172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:20.053673 6172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:20.053714 6172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:20.064903 6172 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36313
I20260927 00:47:20.064925 6300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36313 every 8 connection(s)
I20260927 00:47:20.066223 6172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:47:20.068584 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 6172
I20260927 00:47:20.068722 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:47:20.069072 392 external_mini_cluster.cc:1428] Setting key 248da35f14d2ca1cff8cd1e75795c253
Sep 27 00:47:20 dist-test-slave-86rs krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:47:20.085108 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:20.068679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43363 (local address 127.0.98.62:34549)
0927 00:47:20.068871 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:20.068878 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:20.070192 (+ 1314us) server_negotiation.cc:407] Connection header received
0927 00:47:20.072468 (+ 2276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:20.072472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:20.072540 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:20.072653 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:20.074549 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:20.075163 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:20.076621 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:20.076881 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:20.077001 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:20.080009 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:20.080013 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:20.080016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:20.080045 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:20.082057 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:20.082853 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:20.082861 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:20.082866 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:20.082981 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:20.083500 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:20.083502 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:20.083504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:20.083731 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:20.083814 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:20.084670 (+ 856us) server_negotiation.cc:299] Negotiation successful
0927 00:47:20.084839 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 00:47:20.086328 6303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:20.069298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.2:43363)
0927 00:47:20.069970 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:20.070040 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:20.072091 (+ 2051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:20.072972 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:20.072987 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:20.073685 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:20.074387 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:20.074405 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:20.075305 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:20.075309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:20.076485 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:20.076496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:20.077057 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:20.077061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:20.077273 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:20.078000 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:20.078020 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:20.079763 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:20.082240 (+ 2477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:20.082248 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:20.082268 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:20.082737 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:20.083131 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:20.083136 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:20.083138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:20.083402 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:20.083842 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:20.083853 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:20.084358 (+ 505us) client_negotiation.cc:769] Sending connection context
0927 00:47:20.084635 (+ 277us) client_negotiation.cc:240] Negotiation successful
0927 00:47:20.085134 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":201,"threads_started":1}
I20260927 00:47:20.087947 6301 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34549
I20260927 00:47:20.088359 6301 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:20.089030 6301 heartbeater.cc:511] Master 127.0.98.62:34549 requested a full tablet report, sending...
I20260927 00:47:20.090382 5976 ts_manager.cc:194] Registered new tserver with Master: 11ce427695804d9f8fb5e61296c8c27e (127.0.98.2:36313)
I20260927 00:47:20.091078 5976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:47:20.101866 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34549
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:20.244717 6306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:20.245126 6306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:20.245179 6306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:20.249897 6306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:20.249975 6306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:20.250044 6306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:47:20.256588 6306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:20.258164 6306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:20.259712 6306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:20.272521 6314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:20.272495 6311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:20.272584 6306 server_base.cc:1034] running on GCE node
W20260927 00:47:20.272496 6312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:20.273479 6306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:20.274410 6306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:20.275616 6306 hybrid_clock.cc:648] HybridClock initialized: now 1790470040275588 us; error 44 us; skew 500 ppm
Sep 27 00:47:20 dist-test-slave-86rs krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:20.279026 6306 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:47:20.280429 6306 webserver.cc:460] Webserver started at http://127.0.98.3:33285/ using document root <none> and password file <none>
I20260927 00:47:20.281224 6306 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:20.281323 6306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:20.281685 6306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:20.284829 6306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-2/data/instance:
uuid: "c225acb491b741469b83c9ffcb7a2f22"
format_stamp: "Formatted at 2026-09-27 00:47:20 on dist-test-slave-86rs"
server_key: "63c0a67fc43c68e5f507d3e54cd098dd"
server_key_iv: "82283c1b61f79f3789a277bb2e853e55"
server_key_version: "encryptionkey@0"
I20260927 00:47:20.285604 6306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c225acb491b741469b83c9ffcb7a2f22"
format_stamp: "Formatted at 2026-09-27 00:47:20 on dist-test-slave-86rs"
server_key: "63c0a67fc43c68e5f507d3e54cd098dd"
server_key_iv: "82283c1b61f79f3789a277bb2e853e55"
server_key_version: "encryptionkey@0"
I20260927 00:47:20.290719 6306 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 00:47:20.294028 6321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:20.295758 6306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:47:20.295925 6306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "c225acb491b741469b83c9ffcb7a2f22"
format_stamp: "Formatted at 2026-09-27 00:47:20 on dist-test-slave-86rs"
server_key: "63c0a67fc43c68e5f507d3e54cd098dd"
server_key_iv: "82283c1b61f79f3789a277bb2e853e55"
server_key_version: "encryptionkey@0"
I20260927 00:47:20.296080 6306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:20.311414 6306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:20.315634 6306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:20.315897 6306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:20.316776 6306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:20.317934 6306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:20.318015 6306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:20.318099 6306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:20.318166 6306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:20.329952 6306 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34119
I20260927 00:47:20.329975 6434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34119 every 8 connection(s)
I20260927 00:47:20.331248 6306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:47:20.341343 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 6306
I20260927 00:47:20.341533 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:47:20.341961 392 external_mini_cluster.cc:1428] Setting key 49ea8c55ee1642cfdf2df9cf66fab2f7
Sep 27 00:47:20 dist-test-slave-86rs krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:20.350208 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:20.334272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35491 (local address 127.0.98.62:34549)
0927 00:47:20.334468 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:20.334474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:20.335923 (+ 1449us) server_negotiation.cc:407] Connection header received
0927 00:47:20.337670 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:20.337673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:20.337743 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:20.337838 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:20.339446 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:20.340114 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:20.341732 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:20.342124 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:20.342267 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:20.345721 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:20.345725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:20.345728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:20.345761 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:20.347553 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:20.348261 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:20.348265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:20.348268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:20.348365 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:20.348786 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:20.348790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:20.348792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:20.349034 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:20.349155 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:20.349833 (+ 678us) server_negotiation.cc:299] Negotiation successful
0927 00:47:20.349998 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 00:47:20.351485 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:20.334962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34549 (local address 127.0.98.3:35491)
0927 00:47:20.335726 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:20.335781 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:20.337380 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:20.338066 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:20.338076 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:20.338564 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:20.339248 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:20.339267 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:20.340353 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:20.340356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:20.341563 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:20.341578 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:20.342321 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:20.342325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:20.342517 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:20.343403 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:20.343425 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:20.345497 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:20.347707 (+ 2210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:20.347715 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:20.347742 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:20.348108 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:20.348471 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:20.348476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:20.348478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:20.348672 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:20.349152 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:20.349162 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:20.349535 (+ 373us) client_negotiation.cc:769] Sending connection context
0927 00:47:20.349779 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 00:47:20.350286 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":208,"threads_started":1}
I20260927 00:47:20.353107 6435 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34549
I20260927 00:47:20.353467 6435 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:20.354251 6435 heartbeater.cc:511] Master 127.0.98.62:34549 requested a full tablet report, sending...
I20260927 00:47:20.355860 5976 ts_manager.cc:194] Registered new tserver with Master: c225acb491b741469b83c9ffcb7a2f22 (127.0.98.3:34119)
I20260927 00:47:20.357036 5976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35491
I20260927 00:47:20.357604 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/test-admin.keytab.
Sep 27 00:47:20 dist-test-slave-86rs krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:20.377360 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 00:47:20.381062 6442 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:20.379073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33970 (local address 127.0.98.1:36447)
0927 00:47:20.379466 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:20.379480 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:20.379518 (+ 38us) server_negotiation.cc:407] Connection header received
0927 00:47:20.379602 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:20.379609 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:20.379871 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:20.380100 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:20.380796 (+ 696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33970: BlockingRecv error: recv got EOF from 127.0.0.1:33970 (error 108)
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":115,"threads_started":1}
W20260927 00:47:20.381400 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/krb5cc)
Sep 27 00:47:20 dist-test-slave-86rs krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:20.382875 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 00:47:20 dist-test-slave-86rs krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:20.384490 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 00:47:20 dist-test-slave-86rs krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:47:20.397289 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:20.386012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33984 (local address 127.0.98.1:36447)
0927 00:47:20.386216 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:20.386220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:20.386238 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:47:20.386370 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:20.386372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:20.386424 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:20.386504 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:20.387123 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:20.387988 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:20.389020 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:20.389492 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:20.389589 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:20.392786 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:20.392802 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:20.392824 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:20.392865 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:20.394937 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:20.395549 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:20.395556 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:20.395557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:20.395676 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:20.395989 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:20.395994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:20.395996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:20.396396 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:20.396700 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:20.396895 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 00:47:20.397087 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
W20260927 00:47:20.402460 6442 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:20.401317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33998 (local address 127.0.98.1:36447)
0927 00:47:20.401457 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:20.401461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:20.401643 (+ 182us) server_negotiation.cc:407] Connection header received
0927 00:47:20.401753 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:20.401755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:20.401798 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:20.401870 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:20.402292 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33998: BlockingRecv error: recv got EOF from 127.0.0.1:33998 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 00:47:20.402667 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 00:47:20 dist-test-slave-86rs krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:20.403908 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 00:47:20 dist-test-slave-86rs krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:20.404971 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 00:47:20 dist-test-slave-86rs krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:47:20.415285 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:20.406153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34014 (local address 127.0.98.1:36447)
0927 00:47:20.406308 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:20.406315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:20.406433 (+ 118us) server_negotiation.cc:407] Connection header received
0927 00:47:20.406636 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:20.406640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:20.406704 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:20.406795 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:20.407351 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:20.407981 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:20.408896 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:20.409151 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:20.409255 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:20.411903 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:20.411907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:20.411911 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:20.411947 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:20.413566 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:20.414037 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:20.414039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:20.414041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:20.414142 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:20.414384 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:20.414386 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:20.414387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:20.414598 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:20.414738 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:20.414911 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 00:47:20.415033 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 00:47:20.418712 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790470006085796-392-0/krb5kdc/krb5cc' of type FILE
W20260927 00:47:20.418736 6442 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:20.417535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34016 (local address 127.0.98.1:36447)
0927 00:47:20.417716 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:20.417720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:20.417797 (+ 77us) server_negotiation.cc:407] Connection header received
0927 00:47:20.417909 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:20.417911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:20.417954 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:20.418041 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:20.418588 (+ 547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34016: BlockingRecv error: recv got EOF from 127.0.0.1:34016 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 27 00:47:20 dist-test-slave-86rs krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:20.419988 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 00:47:20 dist-test-slave-86rs krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:20.421041 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 00:47:20 dist-test-slave-86rs krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:47:20.430891 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:20.422063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34022 (local address 127.0.98.1:36447)
0927 00:47:20.422199 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:20.422204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:20.422303 (+ 99us) server_negotiation.cc:407] Connection header received
0927 00:47:20.422458 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:20.422461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:20.422502 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:20.422587 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:20.423126 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:20.423676 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:20.424641 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:20.424861 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:20.424969 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:20.427664 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:20.427668 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:20.427670 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:20.427704 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:20.429154 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:20.429618 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:20.429622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:20.429624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:20.429718 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:20.429994 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:20.429998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:20.430000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:20.430231 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:20.430393 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:20.430596 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 00:47:20.430729 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:47:20.432379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 6038
I20260927 00:47:20.439065 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 6172
I20260927 00:47:20.446252 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 6306
I20260927 00:47:20.452468 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 5944
2026-09-27T00:47:20Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2421 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:47:20 dist-test-slave-86rs krb5kdc[6455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:47:20 dist-test-slave-86rs krb5kdc[6455](info): set up 2 sockets
Sep 27 00:47:20 dist-test-slave-86rs krb5kdc[6455](info): commencing operation
krb5kdc: starting...
W20260927 00:47:21.533265 392 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:47:21 dist-test-slave-86rs krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:47:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:47:21Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 00:47:21.677950 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34945
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:21.813305 6468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:21.813740 6468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:21.813827 6468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:21.818253 6468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:47:21.818367 6468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:21.818406 6468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:21.818447 6468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:47:21.818480 6468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:47:21.824436 6468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:21.826095 6468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:21.827410 6468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:21.838057 6473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:21.838090 6468 server_base.cc:1034] running on GCE node
W20260927 00:47:21.838024 6474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:21.838104 6476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:21.839092 6468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:21.840704 6468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:21.841881 6468 hybrid_clock.cc:648] HybridClock initialized: now 1790470041841866 us; error 38 us; skew 500 ppm
Sep 27 00:47:21 dist-test-slave-86rs krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470041, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:21.852051 6468 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 00:47:21.853402 6468 webserver.cc:460] Webserver started at http://127.0.98.62:45255/ using document root <none> and password file <none>
I20260927 00:47:21.854161 6468 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:21.854262 6468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:21.854550 6468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:21.857337 6468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "5259924bdea44542b5131a31ce919b1d"
format_stamp: "Formatted at 2026-09-27 00:47:21 on dist-test-slave-86rs"
server_key: "fe560661b7344585d00c7befee5e7f95"
server_key_iv: "1edb7ee2645b6700deca1fdfa285656c"
server_key_version: "encryptionkey@0"
I20260927 00:47:21.858026 6468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "5259924bdea44542b5131a31ce919b1d"
format_stamp: "Formatted at 2026-09-27 00:47:21 on dist-test-slave-86rs"
server_key: "fe560661b7344585d00c7befee5e7f95"
server_key_iv: "1edb7ee2645b6700deca1fdfa285656c"
server_key_version: "encryptionkey@0"
I20260927 00:47:21.863065 6468 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 00:47:21.866279 6483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:21.867903 6468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:47:21.868067 6468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "5259924bdea44542b5131a31ce919b1d"
format_stamp: "Formatted at 2026-09-27 00:47:21 on dist-test-slave-86rs"
server_key: "fe560661b7344585d00c7befee5e7f95"
server_key_iv: "1edb7ee2645b6700deca1fdfa285656c"
server_key_version: "encryptionkey@0"
I20260927 00:47:21.868211 6468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:21.883574 6468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:21.887725 6468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:21.887984 6468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:21.896965 6468 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34945
I20260927 00:47:21.897311 6535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34945 every 8 connection(s)
I20260927 00:47:21.898389 6468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:47:21.902186 6536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:21.906834 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 6468
I20260927 00:47:21.906996 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:47:21.907325 392 external_mini_cluster.cc:1428] Setting key d47c2c4b9d1e6faffa2651c5c47455bf
I20260927 00:47:21.910292 6536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d: Bootstrap starting.
I20260927 00:47:21.913712 6536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:21.914706 6536 log.cc:824] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:21.917465 6536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d: No bootstrap required, opened a new log
Sep 27 00:47:21 dist-test-slave-86rs krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:47:21.922199 6536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } }
I20260927 00:47:21.922603 6536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:21.922672 6536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5259924bdea44542b5131a31ce919b1d, State: Initialized, Role: FOLLOWER
I20260927 00:47:21.923343 6536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } }
I20260927 00:47:21.923563 6536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:47:21.923642 6536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:47:21.923789 6536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:21.925163 6536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } }
I20260927 00:47:21.925583 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:21.909218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49776 (local address 127.0.98.62:34945)
0927 00:47:21.909764 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:21.909784 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:21.909842 (+ 58us) server_negotiation.cc:407] Connection header received
0927 00:47:21.910677 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:21.910705 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:21.911174 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:21.911555 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:21.912672 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:21.913899 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:21.914951 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:21.915599 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:21.915810 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:21.918890 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:21.918896 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:21.918914 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:21.918946 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:21.921458 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:21.922182 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:21.922189 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:21.922195 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:21.922340 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:21.922707 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:21.922713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:21.922717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:21.923381 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:21.923642 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:21.924016 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 00:47:21.924443 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":165,"threads_started":1}
I20260927 00:47:21.925797 6536 leader_election.cc:302] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5259924bdea44542b5131a31ce919b1d; no voters:
I20260927 00:47:21.926307 6536 leader_election.cc:288] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:47:21.926442 6541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:21.926698 6541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [term 1 LEADER]: Becoming Leader. State: Replica: 5259924bdea44542b5131a31ce919b1d, State: Running, Role: LEADER
I20260927 00:47:21.927095 6541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } }
I20260927 00:47:21.927932 6536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:47:21.929349 6543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5259924bdea44542b5131a31ce919b1d. Latest consensus state: current_term: 1 leader_uuid: "5259924bdea44542b5131a31ce919b1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } }
I20260927 00:47:21.929527 6543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:21.929489 6542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5259924bdea44542b5131a31ce919b1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } }
I20260927 00:47:21.929586 6542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:21.930035 6550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:47:21.934173 6550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:47:21.941924 6550 catalog_manager.cc:1347] Generated new cluster ID: 6864942181e149ff81377ee6995feab8
I20260927 00:47:21.942028 6550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:47:21.956620 6550 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:47:21.957885 6550 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:47:21.963243 6550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d: Generated new TSK 0
I20260927 00:47:21.964658 6550 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 00:47:22.005944 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34945
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:22.146471 6562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:22.146838 6562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:22.146888 6562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:22.151300 6562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:22.151413 6562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:22.151484 6562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:47:22.157903 6562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:22.159556 6562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:22.161113 6562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:47:22.174285 6562 server_base.cc:1034] running on GCE node
W20260927 00:47:22.174228 6570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:22.174225 6568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:22.174232 6567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:22.175102 6562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:22.176007 6562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:22.177235 6562 hybrid_clock.cc:648] HybridClock initialized: now 1790470042177229 us; error 80 us; skew 500 ppm
Sep 27 00:47:22 dist-test-slave-86rs krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470042, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:22.181214 6562 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 00:47:22.182569 6562 webserver.cc:460] Webserver started at http://127.0.98.1:33817/ using document root <none> and password file <none>
I20260927 00:47:22.183413 6562 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:22.183528 6562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:22.183876 6562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:22.186856 6562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/data/instance:
uuid: "8ed31e81e1864605b6c77a8123fdb062"
format_stamp: "Formatted at 2026-09-27 00:47:22 on dist-test-slave-86rs"
server_key: "eb0238b16014acf6238db01a93e1ffc1"
server_key_iv: "e1abd7e76eee7e9995ce14d6bf364cab"
server_key_version: "encryptionkey@0"
I20260927 00:47:22.187645 6562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8ed31e81e1864605b6c77a8123fdb062"
format_stamp: "Formatted at 2026-09-27 00:47:22 on dist-test-slave-86rs"
server_key: "eb0238b16014acf6238db01a93e1ffc1"
server_key_iv: "e1abd7e76eee7e9995ce14d6bf364cab"
server_key_version: "encryptionkey@0"
I20260927 00:47:22.192937 6562 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 00:47:22.196302 6577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:22.197988 6562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:47:22.198179 6562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "8ed31e81e1864605b6c77a8123fdb062"
format_stamp: "Formatted at 2026-09-27 00:47:22 on dist-test-slave-86rs"
server_key: "eb0238b16014acf6238db01a93e1ffc1"
server_key_iv: "e1abd7e76eee7e9995ce14d6bf364cab"
server_key_version: "encryptionkey@0"
I20260927 00:47:22.198333 6562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:22.216470 6562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:22.220757 6562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:22.221060 6562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:22.221939 6562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:22.223114 6562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:22.223196 6562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:22.223284 6562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:22.223327 6562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:22.235193 6562 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41565
I20260927 00:47:22.235249 6690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41565 every 8 connection(s)
I20260927 00:47:22.236490 6562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:47:22.245386 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 6562
I20260927 00:47:22.245579 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:47:22.245973 392 external_mini_cluster.cc:1428] Setting key c128129b4a3e86dc09a79a30b9cbd5eb
Sep 27 00:47:22 dist-test-slave-86rs krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470042, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:47:22.256345 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:22.239140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55607 (local address 127.0.98.62:34945)
0927 00:47:22.239332 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:22.239338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:22.240803 (+ 1465us) server_negotiation.cc:407] Connection header received
0927 00:47:22.242465 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:22.242469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:22.242536 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:22.242627 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:22.244437 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:22.245051 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:22.246972 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:22.247292 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:22.247403 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:22.251357 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:22.251361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:22.251363 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:22.251395 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:22.253351 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:22.254166 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:22.254170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:22.254171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:22.254268 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:22.254778 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:22.254780 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:22.254781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:22.255000 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:22.255078 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:22.255976 (+ 898us) server_negotiation.cc:299] Negotiation successful
0927 00:47:22.256115 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 00:47:22.258009 6693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:22.239756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.98.1:55607)
0927 00:47:22.240557 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:22.240621 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:22.242153 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:22.242946 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:22.242959 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:22.243499 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:22.244168 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:22.244187 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:22.245230 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:22.245237 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:22.246811 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:22.246822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:22.247473 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:22.247477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:22.247757 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:22.248643 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:22.248673 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:22.251110 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:22.253503 (+ 2393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:22.253512 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:22.253539 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:22.254025 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:22.254381 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:22.254387 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:22.254392 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:22.254661 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:22.255117 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:22.255130 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:22.255625 (+ 495us) client_negotiation.cc:769] Sending connection context
0927 00:47:22.255918 (+ 293us) client_negotiation.cc:240] Negotiation successful
0927 00:47:22.256463 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":160,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:47:22.259717 6691 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34945
I20260927 00:47:22.260051 6691 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:22.260854 6691 heartbeater.cc:511] Master 127.0.98.62:34945 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260927 00:47:22.263052 6500 ts_manager.cc:194] Registered new tserver with Master: 8ed31e81e1864605b6c77a8123fdb062 (127.0.98.1:41565)
I20260927 00:47:22.264892 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55607
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 00:47:22.280038 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34945
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:22.417747 6696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:22.418138 6696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:22.418215 6696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:22.422572 6696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:22.422688 6696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:22.422791 6696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:47:22.429064 6696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:22.430761 6696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:22.432301 6696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:22.445013 6701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:22.445109 6696 server_base.cc:1034] running on GCE node
W20260927 00:47:22.445014 6702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:22.445014 6704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:22.445919 6696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:22.446758 6696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:22.447952 6696 hybrid_clock.cc:648] HybridClock initialized: now 1790470042447934 us; error 55 us; skew 500 ppm
Sep 27 00:47:22 dist-test-slave-86rs krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470042, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:22.451076 6696 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 00:47:22.452114 6696 webserver.cc:460] Webserver started at http://127.0.98.2:44893/ using document root <none> and password file <none>
I20260927 00:47:22.452867 6696 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:22.452934 6696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:22.453141 6696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:22.455590 6696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/data/instance:
uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86"
format_stamp: "Formatted at 2026-09-27 00:47:22 on dist-test-slave-86rs"
server_key: "dc4c825a77bd767bf6f7f95199b42c91"
server_key_iv: "3c4f52435c646bc0ff3252dac0488e37"
server_key_version: "encryptionkey@0"
I20260927 00:47:22.456207 6696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86"
format_stamp: "Formatted at 2026-09-27 00:47:22 on dist-test-slave-86rs"
server_key: "dc4c825a77bd767bf6f7f95199b42c91"
server_key_iv: "3c4f52435c646bc0ff3252dac0488e37"
server_key_version: "encryptionkey@0"
I20260927 00:47:22.460932 6696 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:47:22.464131 6711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:22.465951 6696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:47:22.466105 6696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86"
format_stamp: "Formatted at 2026-09-27 00:47:22 on dist-test-slave-86rs"
server_key: "dc4c825a77bd767bf6f7f95199b42c91"
server_key_iv: "3c4f52435c646bc0ff3252dac0488e37"
server_key_version: "encryptionkey@0"
I20260927 00:47:22.466274 6696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:22.483742 6696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:22.488021 6696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:22.488300 6696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:22.489073 6696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:22.490099 6696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:22.490157 6696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:22.490207 6696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:22.490223 6696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:22.501428 6696 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34123
I20260927 00:47:22.501464 6824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34123 every 8 connection(s)
I20260927 00:47:22.502647 6696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:47:22.508503 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 6696
I20260927 00:47:22.508651 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:47:22.508988 392 external_mini_cluster.cc:1428] Setting key f666a8705d975c51dcddd37bb39e06bb
Sep 27 00:47:22 dist-test-slave-86rs krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470042, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:47:22.521354 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:22.505100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36709 (local address 127.0.98.62:34945)
0927 00:47:22.505301 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:22.505306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:22.506436 (+ 1130us) server_negotiation.cc:407] Connection header received
0927 00:47:22.508024 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:22.508027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:22.508113 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:22.508197 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:22.510193 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:22.511020 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:22.513037 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:22.513334 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:22.513461 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:22.516894 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:22.516902 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:22.516907 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:22.516948 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:22.518612 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:22.519384 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:22.519388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:22.519390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:22.519476 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:22.519901 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:22.519903 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:22.519904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:22.520109 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:22.520193 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:22.521036 (+ 843us) server_negotiation.cc:299] Negotiation successful
0927 00:47:22.521198 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:47:22.522917 6827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:22.505696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.98.2:36709)
0927 00:47:22.506245 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:22.506298 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:22.507766 (+ 1468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:22.508538 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:22.508551 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:22.509171 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:22.510011 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:22.510033 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:22.511165 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:22.511169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:22.512850 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:22.512870 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:22.513525 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:22.513530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:22.513751 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:22.514543 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:22.514564 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:22.516642 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:22.518751 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:22.518762 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:22.518790 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:22.519265 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:22.519565 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:22.519570 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:22.519572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:22.519795 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:22.520251 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:22.520261 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:22.520709 (+ 448us) client_negotiation.cc:769] Sending connection context
0927 00:47:22.520993 (+ 284us) client_negotiation.cc:240] Negotiation successful
0927 00:47:22.521505 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":141,"threads_started":1}
I20260927 00:47:22.524648 6825 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34945
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260927 00:47:22.525066 6825 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:22.525766 6825 heartbeater.cc:511] Master 127.0.98.62:34945 requested a full tablet report, sending...
I20260927 00:47:22.527164 6500 ts_manager.cc:194] Registered new tserver with Master: 1fedc53da95e4fc5bcdc6e2a3ee28a86 (127.0.98.2:34123)
I20260927 00:47:22.527987 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36709
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 00:47:22.541716 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34945
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:22.680133 6830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:22.680521 6830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:22.680575 6830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:22.684885 6830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:22.684960 6830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:22.685029 6830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:47:22.691295 6830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:22.692785 6830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:22.694114 6830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:22.706408 6838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:22.706437 6835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:22.706408 6836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:22.706835 6830 server_base.cc:1034] running on GCE node
I20260927 00:47:22.707401 6830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:22.708386 6830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:22.709586 6830 hybrid_clock.cc:648] HybridClock initialized: now 1790470042709560 us; error 38 us; skew 500 ppm
Sep 27 00:47:22 dist-test-slave-86rs krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470042, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:22.713016 6830 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 00:47:22.714177 6830 webserver.cc:460] Webserver started at http://127.0.98.3:44215/ using document root <none> and password file <none>
I20260927 00:47:22.714908 6830 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:22.715006 6830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:22.715288 6830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:22.717985 6830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/data/instance:
uuid: "f4312a0972ce4ab3905b603a432eacb8"
format_stamp: "Formatted at 2026-09-27 00:47:22 on dist-test-slave-86rs"
server_key: "e03f7bdb9ae77d332406ca54b3fe0974"
server_key_iv: "b81f599661f83f20d1de4117e1454b03"
server_key_version: "encryptionkey@0"
I20260927 00:47:22.718681 6830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f4312a0972ce4ab3905b603a432eacb8"
format_stamp: "Formatted at 2026-09-27 00:47:22 on dist-test-slave-86rs"
server_key: "e03f7bdb9ae77d332406ca54b3fe0974"
server_key_iv: "b81f599661f83f20d1de4117e1454b03"
server_key_version: "encryptionkey@0"
I20260927 00:47:22.723399 6830 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:47:22.726604 6845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:22.728379 6830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 00:47:22.728610 6830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "f4312a0972ce4ab3905b603a432eacb8"
format_stamp: "Formatted at 2026-09-27 00:47:22 on dist-test-slave-86rs"
server_key: "e03f7bdb9ae77d332406ca54b3fe0974"
server_key_iv: "b81f599661f83f20d1de4117e1454b03"
server_key_version: "encryptionkey@0"
I20260927 00:47:22.728776 6830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:22.749881 6830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:22.754132 6830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:22.754365 6830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:22.755125 6830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:22.756179 6830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:22.756319 6830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:22.756383 6830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:22.756400 6830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:22.767666 6830 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42375
I20260927 00:47:22.767686 6958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42375 every 8 connection(s)
I20260927 00:47:22.768960 6830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:47:22.770738 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 6830
I20260927 00:47:22.770885 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:47:22.771267 392 external_mini_cluster.cc:1428] Setting key ca1551f1b0cd57190e2ce07e99d4235e
Sep 27 00:47:22 dist-test-slave-86rs krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470042, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:47:22.786958 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:22.771595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59993 (local address 127.0.98.62:34945)
0927 00:47:22.771790 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:22.771795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:22.773257 (+ 1462us) server_negotiation.cc:407] Connection header received
0927 00:47:22.775385 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:22.775388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:22.775455 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:22.775535 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:22.777100 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:22.777705 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:22.778994 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:22.779238 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:22.779356 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:22.782259 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:22.782262 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:22.782265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:22.782296 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:22.783706 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:22.784615 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:22.784635 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:22.784637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:22.784753 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:22.785217 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:22.785221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:22.785222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:22.785540 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:22.785630 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:22.786537 (+ 907us) server_negotiation.cc:299] Negotiation successful
0927 00:47:22.786754 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 00:47:22.788030 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:22.772324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.98.3:59993)
0927 00:47:22.773037 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:22.773136 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:22.775125 (+ 1989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:22.775761 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:22.775770 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:22.776270 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:22.776944 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:22.776960 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:22.777843 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:22.777846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:22.778860 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:22.778869 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:22.779408 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:22.779413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:22.779630 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:22.780362 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:22.780382 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:22.782058 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:22.783831 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:22.783836 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:22.783854 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:22.784214 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:22.784857 (+ 643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:22.784862 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:22.784864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:22.785054 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:22.785657 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:22.785666 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:22.786065 (+ 399us) client_negotiation.cc:769] Sending connection context
0927 00:47:22.786375 (+ 310us) client_negotiation.cc:240] Negotiation successful
0927 00:47:22.786863 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":189,"threads_started":1}
I20260927 00:47:22.789649 6959 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34945
I20260927 00:47:22.790016 6959 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:22.790681 6959 heartbeater.cc:511] Master 127.0.98.62:34945 requested a full tablet report, sending...
I20260927 00:47:22.792050 6500 ts_manager.cc:194] Registered new tserver with Master: f4312a0972ce4ab3905b603a432eacb8 (127.0.98.3:42375)
I20260927 00:47:22.792819 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59993
I20260927 00:47:22.799033 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260927 00:47:22.833053 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34013
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34945,127.0.98.61:34013
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:22.983909 6964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:22.984400 6964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:22.984457 6964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:22.989133 6964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:47:22.989244 6964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:22.989266 6964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:22.989285 6964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:47:22.989301 6964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:47:22.995726 6964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34945,127.0.98.61:34013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
W20260927 00:47:22.996066 6964 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:34945,127.0.98.61:34013'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 00:47:22.997365 6964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:22.998735 6964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:23.009640 6970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:23.009644 6972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:23.009647 6969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:23.010025 6964 server_base.cc:1034] running on GCE node
I20260927 00:47:23.010802 6964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:23.012424 6964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:23.013604 6964 hybrid_clock.cc:648] HybridClock initialized: now 1790470043013570 us; error 52 us; skew 500 ppm
Sep 27 00:47:23 dist-test-slave-86rs krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470043, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:23.017109 6964 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 00:47:23.018314 6964 webserver.cc:460] Webserver started at http://127.0.98.61:38687/ using document root <none> and password file <none>
I20260927 00:47:23.019079 6964 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:23.019182 6964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:23.019466 6964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:23.022286 6964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-1/data/instance:
uuid: "ceb92bf11e1b41a19b0118bb8a88002d"
format_stamp: "Formatted at 2026-09-27 00:47:23 on dist-test-slave-86rs"
server_key: "f75a9100a2d79d80f52f32cde0b14932"
server_key_iv: "0b4dab98a73d6e6c4968179e583204ff"
server_key_version: "encryptionkey@0"
I20260927 00:47:23.023010 6964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-1/wal/instance:
uuid: "ceb92bf11e1b41a19b0118bb8a88002d"
format_stamp: "Formatted at 2026-09-27 00:47:23 on dist-test-slave-86rs"
server_key: "f75a9100a2d79d80f52f32cde0b14932"
server_key_iv: "0b4dab98a73d6e6c4968179e583204ff"
server_key_version: "encryptionkey@0"
I20260927 00:47:23.028209 6964 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 00:47:23.031776 6979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:23.033826 6964 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:47:23.033994 6964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-1/wal
uuid: "ceb92bf11e1b41a19b0118bb8a88002d"
format_stamp: "Formatted at 2026-09-27 00:47:23 on dist-test-slave-86rs"
server_key: "f75a9100a2d79d80f52f32cde0b14932"
server_key_iv: "0b4dab98a73d6e6c4968179e583204ff"
server_key_version: "encryptionkey@0"
I20260927 00:47:23.034147 6964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:23.055075 6964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:23.059322 6964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:23.059583 6964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:23.069934 6964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34013
I20260927 00:47:23.070817 7031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34013 every 8 connection(s)
I20260927 00:47:23.072028 6964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-1/data/info.pb
I20260927 00:47:23.072579 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 6964
I20260927 00:47:23.076333 7032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:23.081241 7032 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:47:23 dist-test-slave-86rs krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 00:47:23 dist-test-slave-86rs krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470043, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:47:23.090317 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.074599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34170 (local address 127.0.98.61:34013)
0927 00:47:23.075114 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:23.075141 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:23.075432 (+ 291us) server_negotiation.cc:407] Connection header received
0927 00:47:23.076572 (+ 1140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:23.076597 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:23.077054 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:23.077484 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:23.078366 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.079507 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.080515 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.081087 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.081276 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.084485 (+ 3209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:23.084492 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:23.084511 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:23.084547 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:23.086540 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.087078 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.087083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.087089 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.087220 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.087528 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.087531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.087533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.088132 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:23.088446 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:23.088805 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 00:47:23.089193 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":158,"threads_started":1}
I20260927 00:47:23.094478 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.083654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49784 (local address 127.0.98.62:34945)
0927 00:47:23.083846 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:23.083852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:23.084394 (+ 542us) server_negotiation.cc:407] Connection header received
0927 00:47:23.084625 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:23.084629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:23.084697 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:23.084789 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:23.085541 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.086133 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.087356 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.087685 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.087805 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.090547 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:23.090551 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:23.090554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:23.090591 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:23.092373 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.092911 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.092915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.092917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.093026 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.093433 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.093437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.093439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.093779 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:23.093869 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:23.094202 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 00:47:23.094331 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 00:47:23.094549 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.083557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.0.1:49784)
0927 00:47:23.084191 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:23.084310 (+ 119us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:23.084507 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:23.084823 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:23.084829 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:23.085250 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:23.085394 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.085401 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.086253 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.086258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.087213 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.087223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.087832 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.087836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.087950 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.088530 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:23.088547 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:23.090329 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:23.092519 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.092527 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.092533 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.092796 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.093145 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.093148 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.093150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.093320 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.093878 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:23.093883 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:23.094011 (+ 128us) client_negotiation.cc:769] Sending connection context
0927 00:47:23.094141 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 00:47:23.094343 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":227,"threads_started":1}
I20260927 00:47:23.095991 7032 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:47:23 dist-test-slave-86rs krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470043, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 00:47:23.106364 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.096633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34013 (local address 127.0.0.1:34180)
0927 00:47:23.096773 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:23.096788 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:23.096869 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:23.097066 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:23.097068 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:23.097243 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:23.097368 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.097374 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.098150 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.098153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.098718 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.098726 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.099174 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.099177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.099250 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.099880 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:23.099892 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:23.102357 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:23.104599 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.104602 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.104604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.104840 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.105185 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.105188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.105189 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.105274 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.105825 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:23.105828 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:23.105928 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 00:47:23.106027 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 00:47:23.106192 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 00:47:23.106482 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.096766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34180 (local address 127.0.98.61:34013)
0927 00:47:23.096884 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:23.096888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:23.096901 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:47:23.096932 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:23.096934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:23.096969 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:23.097034 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:23.097474 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.098039 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.098828 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.099061 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.099167 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.102536 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:23.102540 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:23.102543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:23.102574 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:23.104463 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.104980 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.104983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.104985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.105072 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.105389 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.105392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.105394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.105710 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:23.105816 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:23.106184 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 00:47:23.106307 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:47:23.110855 7032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d: Bootstrap starting.
I20260927 00:47:23.114308 7032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:23.115749 7032 log.cc:824] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:23.118587 7032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d: No bootstrap required, opened a new log
I20260927 00:47:23.122406 7032 raft_consensus.cc:348] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } }
I20260927 00:47:23.122699 7032 raft_consensus.cc:374] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:23.122740 7032 raft_consensus.cc:729] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb92bf11e1b41a19b0118bb8a88002d, State: Initialized, Role: FOLLOWER
I20260927 00:47:23.123257 7032 consensus_queue.cc:260] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } }
I20260927 00:47:23.123781 7038 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } } }
I20260927 00:47:23.123950 7038 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:47:23.124902 7032 sys_catalog.cc:564] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [sys.catalog]: configured and running, proceeding with master startup.
W20260927 00:47:23.130975 7049 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:47:23.131111 7049 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:47:23.133285 6964 master_runner.cc:428] Detected that this master ceb92bf11e1b41a19b0118bb8a88002d is joining an existing cluster
I20260927 00:47:23.133380 6964 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:34013
I20260927 00:47:23.146541 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.136653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34188 (local address 127.0.98.61:34013)
0927 00:47:23.136768 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:23.136772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:23.138581 (+ 1809us) server_negotiation.cc:407] Connection header received
0927 00:47:23.138722 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:23.138726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:23.138780 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:23.138848 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:23.139388 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.140175 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.141371 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.141714 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.141821 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.143117 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:23.143121 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:23.143124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:23.143151 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:23.144813 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.145240 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.145247 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.145249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.145332 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.145598 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.145604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.145606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.145811 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:23.145925 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:23.146163 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 00:47:23.146332 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":48}
I20260927 00:47:23.146556 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.136762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34013 (local address 127.0.0.1:34188)
0927 00:47:23.138477 (+ 1715us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:23.138489 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:23.138613 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:23.138947 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:23.138949 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:23.139176 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:23.139264 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.139269 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.140334 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.140339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.141226 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.141240 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.141846 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.141852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.141963 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.142487 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:23.142498 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:23.142982 (+ 484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:23.144916 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.144919 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.144921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.145144 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.145420 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.145425 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.145427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.145512 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.145910 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:23.145913 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:23.145999 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 00:47:23.146111 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 00:47:23.146388 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1648,"mutex_wait_us":113,"thread_start_us":97,"threads_started":1}
I20260927 00:47:23.149791 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.136004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.0.1:49800)
0927 00:47:23.136409 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:23.136426 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:23.136562 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:23.136883 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:23.136886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:23.137168 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:23.137357 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.137366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.138199 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.138204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.139294 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.139305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.140077 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.140080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.140201 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.141378 (+ 1177us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:23.141395 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:23.141823 (+ 428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:23.144550 (+ 2727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.144553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.144556 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.144903 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.148451 (+ 3548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.148455 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.148458 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.148568 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.149135 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:23.149139 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:23.149250 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 00:47:23.149352 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 00:47:23.149550 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":3,"thread_start_us":146,"threads_started":1}
I20260927 00:47:23.149868 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.136164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49800 (local address 127.0.98.62:34945)
0927 00:47:23.136371 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:23.136379 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:23.136534 (+ 155us) server_negotiation.cc:407] Connection header received
0927 00:47:23.136667 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:23.136673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:23.136755 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:23.136863 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:23.137481 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.138090 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.139464 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.139823 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.139926 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.142015 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:23.142022 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:23.142026 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:23.142061 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:23.144153 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.148147 (+ 3994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.148153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.148158 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.148339 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.148696 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.148701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.148706 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.149022 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:23.149143 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:23.149458 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 00:47:23.149642 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Sep 27 00:47:23 dist-test-slave-86rs krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470041, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 00:47:23.170405 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.156919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34194 (local address 127.0.98.61:34013)
0927 00:47:23.157107 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:23.157112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:23.157694 (+ 582us) server_negotiation.cc:407] Connection header received
0927 00:47:23.158066 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:23.158069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:23.158171 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:23.158276 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:23.159492 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.160402 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.161686 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.162066 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.162197 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.165735 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:23.165740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:23.165743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:23.165784 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:23.167722 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.168570 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.168574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.168576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.168686 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.169061 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.169064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.169066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.169346 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:23.169451 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:23.170023 (+ 572us) server_negotiation.cc:299] Negotiation successful
0927 00:47:23.170194 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 00:47:23.170532 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.156880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34013 (local address 127.0.0.1:34194)
0927 00:47:23.157498 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:23.157578 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:23.157852 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:23.158363 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:23.158371 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:23.159024 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:23.159271 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.159285 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.160579 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.160583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.161500 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.161552 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.162254 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.162261 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.162461 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.163303 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:23.163319 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:23.165475 (+ 2156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:23.167897 (+ 2422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.167917 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.167931 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.168433 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.168805 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.168808 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.168811 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.168950 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.169482 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:23.169496 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:23.169730 (+ 234us) client_negotiation.cc:769] Sending connection context
0927 00:47:23.169952 (+ 222us) client_negotiation.cc:240] Negotiation successful
0927 00:47:23.170288 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":136,"threads_started":1}
I20260927 00:47:23.172734 6500 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:34013: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 34013
}
attrs {
promote: true
}
}
dest_uuid: "5259924bdea44542b5131a31ce919b1d"
cas_config_opid_index: -1
I20260927 00:47:23.173761 6500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: true } }
I20260927 00:47:23.176772 7061 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ceb92bf11e1b41a19b0118bb8a88002d (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: true } } }
I20260927 00:47:23.177444 7006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:23.180022 7061 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:34013
I20260927 00:47:23.180052 7006 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [term 1 FOLLOWER]: Refusing update from remote peer 5259924bdea44542b5131a31ce919b1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 00:47:23.180835 6964 master.cc:537] Master@127.0.98.61:34013 shutting down...
I20260927 00:47:23.180616 7061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5259924bdea44542b5131a31ce919b1d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: true } } }
I20260927 00:47:23.181124 7061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:23.181272 7062 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.60.keytab.
W20260927 00:47:23.184386 6485 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260927 00:47:23.185014 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:32877
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34945,127.0.98.61:34013,127.0.98.60:32877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:23.189548 6485 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d -> Peer ceb92bf11e1b41a19b0118bb8a88002d (127.0.98.61:34013): Couldn't send request to peer ceb92bf11e1b41a19b0118bb8a88002d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 00:47:23.190346 6964 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:47:23.190806 6964 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:23.190940 6964 tablet_replica.cc:330] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d: stopping tablet replica
I20260927 00:47:23.207875 6964 master.cc:552] Master@127.0.98.61:34013 shutdown complete.
I20260927 00:47:23.208025 6964 master_runner.cc:305] Clearing existing system tablet
I20260927 00:47:23.208432 6964 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:23.208503 6964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:23.210228 6964 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 00:47:23.211359 7067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:23.211678 6964 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:23.211824 6964 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 00:47:23.211864 6964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-1/wal
uuid: "ceb92bf11e1b41a19b0118bb8a88002d"
format_stamp: "Formatted at 2026-09-27 00:47:23 on dist-test-slave-86rs"
server_key: "f75a9100a2d79d80f52f32cde0b14932"
server_key_iv: "0b4dab98a73d6e6c4968179e583204ff"
server_key_version: "encryptionkey@0"
I20260927 00:47:23.214831 6964 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:23.218386 6964 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 00:47:23.218550 6964 log.cc:1196] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 00:47:23.218875 6964 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d: Deleting consensus metadata
I20260927 00:47:23.219087 6964 master_runner.cc:315] Copying system tablet from 127.0.98.62:34945
I20260927 00:47:23.220707 6964 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:34945
I20260927 00:47:23.232707 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.221675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49804 (local address 127.0.98.62:34945)
0927 00:47:23.221860 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:23.221867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:23.222027 (+ 160us) server_negotiation.cc:407] Connection header received
0927 00:47:23.222195 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:23.222199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:23.222271 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:23.222379 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:23.223093 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.224036 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.225472 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.225850 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.225998 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.227967 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:23.227972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:23.227976 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:23.228015 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:23.230257 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.230867 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.230872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.230874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.231078 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.231478 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.231483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.231485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.231800 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:23.231926 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:23.232312 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 00:47:23.232485 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 00:47:23.233366 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.221515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.0.1:49804)
0927 00:47:23.221863 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:23.221884 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:23.222069 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:23.222418 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:23.222421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:23.222712 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:23.222903 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.222913 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.224178 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.224182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.225310 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.225327 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.226322 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.226327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.226439 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.227189 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:23.227207 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:23.227746 (+ 539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:23.230439 (+ 2693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.230443 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.230446 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.230747 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.231233 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.231237 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.231239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.231348 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.231917 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:23.231921 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:23.232056 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 00:47:23.232186 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 00:47:23.232426 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":127,"threads_started":1}
I20260927 00:47:23.234139 6520 tablet_copy_service.cc:140] P 5259924bdea44542b5131a31ce919b1d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ceb92bf11e1b41a19b0118bb8a88002d ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:49804)
I20260927 00:47:23.234402 6520 tablet_copy_service.cc:161] P 5259924bdea44542b5131a31ce919b1d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ceb92bf11e1b41a19b0118bb8a88002d at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:49804: session id = ceb92bf11e1b41a19b0118bb8a88002d-00000000000000000000000000000000
I20260927 00:47:23.237192 6520 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 00:47:23.239935 6964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 00:47:23.243388 6964 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d: tablet copy: Starting download of 0 data blocks...
I20260927 00:47:23.243646 6964 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d: tablet copy: Starting download of 1 WAL segments...
I20260927 00:47:23.246167 6964 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 00:47:23.248803 6520 tablet_copy_service.cc:342] P 5259924bdea44542b5131a31ce919b1d: Request end of tablet copy session ceb92bf11e1b41a19b0118bb8a88002d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:49804
I20260927 00:47:23.249022 6520 tablet_copy_service.cc:433] P 5259924bdea44542b5131a31ce919b1d: ending tablet copy session ceb92bf11e1b41a19b0118bb8a88002d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ceb92bf11e1b41a19b0118bb8a88002d
I20260927 00:47:23.259725 6964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:23.262066 7074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:23.262131 7077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:23.262686 6964 server_base.cc:1034] running on GCE node
W20260927 00:47:23.262825 7075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:23.263140 6964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:23.263556 6964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:23.264710 6964 hybrid_clock.cc:648] HybridClock initialized: now 1790470043264695 us; error 43 us; skew 500 ppm
I20260927 00:47:23.268657 6691 heartbeater.cc:503] Master 127.0.98.62:34945 was elected leader, sending a full tablet report...
Sep 27 00:47:23 dist-test-slave-86rs krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470043, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:23.271682 6964 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 00:47:23.272413 6964 webserver.cc:460] Webserver started at http://127.0.98.61:36349/ using document root <none> and password file <none>
I20260927 00:47:23.272643 6964 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:23.272696 6964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:23.274515 6964 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 00:47:23.275808 7084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:23.276118 6964 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 00:47:23.276209 6964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-1/wal
uuid: "ceb92bf11e1b41a19b0118bb8a88002d"
format_stamp: "Formatted at 2026-09-27 00:47:23 on dist-test-slave-86rs"
server_key: "f75a9100a2d79d80f52f32cde0b14932"
server_key_iv: "0b4dab98a73d6e6c4968179e583204ff"
server_key_version: "encryptionkey@0"
I20260927 00:47:23.276338 6964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:23.305363 6964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:23.307859 6964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:23.313769 6964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34013
I20260927 00:47:23.313820 7136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34013 every 8 connection(s)
I20260927 00:47:23.314474 6964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-1/data/info.pb
I20260927 00:47:23.318281 7137 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:47:23.319621 7137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d: Bootstrap starting.
I20260927 00:47:23.335350 7137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:47:23.335795 7137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d: Bootstrap complete.
I20260927 00:47:23.336588 7137 raft_consensus.cc:348] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: true } }
I20260927 00:47:23.336776 7137 raft_consensus.cc:729] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ceb92bf11e1b41a19b0118bb8a88002d, State: Initialized, Role: LEARNER
I20260927 00:47:23.336884 7137 consensus_queue.cc:260] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: true } }
I20260927 00:47:23.337340 7140 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: true } } }
I20260927 00:47:23.337459 7137 sys_catalog.cc:564] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:47:23.337606 7140 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [sys.catalog]: This master's current role is: LEARNER
I20260927 00:47:23.341609 7151 catalog_manager.cc:1259] Loaded cluster ID: 6864942181e149ff81377ee6995feab8
I20260927 00:47:23.341840 7151 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d: loading cluster ID for follower catalog manager: success
I20260927 00:47:23.344404 7151 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d: acquiring CA information for follower catalog manager: success
I20260927 00:47:23.346248 7151 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 00:47:23.362847 7065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:23.363250 7065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:23.363299 7065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:23.367655 7065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:47:23.367758 7065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:23.367779 7065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:23.367796 7065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:47:23.367812 7065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:47:23.374629 7065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34945,127.0.98.61:34013,127.0.98.60:32877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:32877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:23.376106 7065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:23.377710 7065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:23.390355 7159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:23.390355 7157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:23.390355 7156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:23.390703 7065 server_base.cc:1034] running on GCE node
I20260927 00:47:23.391599 7065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:23.393241 7065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:23.394479 7065 hybrid_clock.cc:648] HybridClock initialized: now 1790470043394399 us; error 76 us; skew 500 ppm
Sep 27 00:47:23 dist-test-slave-86rs krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470043, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:23.398185 7065 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 00:47:23.399489 7065 webserver.cc:460] Webserver started at http://127.0.98.60:34599/ using document root <none> and password file <none>
I20260927 00:47:23.400353 7065 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:23.400476 7065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:23.400787 7065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:23.403869 7065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-2/data/instance:
uuid: "e3bfc9d2ca344168be147ebe68ef5d31"
format_stamp: "Formatted at 2026-09-27 00:47:23 on dist-test-slave-86rs"
server_key: "f6548a99562ae8e1b0f02f3812bb1d83"
server_key_iv: "28597a09fa36edff90cce049404fe745"
server_key_version: "encryptionkey@0"
I20260927 00:47:23.404685 7065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-2/wal/instance:
uuid: "e3bfc9d2ca344168be147ebe68ef5d31"
format_stamp: "Formatted at 2026-09-27 00:47:23 on dist-test-slave-86rs"
server_key: "f6548a99562ae8e1b0f02f3812bb1d83"
server_key_iv: "28597a09fa36edff90cce049404fe745"
server_key_version: "encryptionkey@0"
I20260927 00:47:23.409912 7065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 00:47:23.413023 7166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:23.414613 7065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:47:23.414790 7065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-2/wal
uuid: "e3bfc9d2ca344168be147ebe68ef5d31"
format_stamp: "Formatted at 2026-09-27 00:47:23 on dist-test-slave-86rs"
server_key: "f6548a99562ae8e1b0f02f3812bb1d83"
server_key_iv: "28597a09fa36edff90cce049404fe745"
server_key_version: "encryptionkey@0"
I20260927 00:47:23.414929 7065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:23.449913 7065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:23.453764 7065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:23.454003 7065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:23.463749 7065 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:32877
I20260927 00:47:23.464068 7218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:32877 every 8 connection(s)
I20260927 00:47:23.465334 7065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-2/data/info.pb
I20260927 00:47:23.468498 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 7065
I20260927 00:47:23.469291 7219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:23.475257 7219 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:47:23 dist-test-slave-86rs krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 00:47:23 dist-test-slave-86rs krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470043, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:47:23.486778 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.470417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37590 (local address 127.0.98.60:32877)
0927 00:47:23.470981 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:23.471001 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:23.471047 (+ 46us) server_negotiation.cc:407] Connection header received
0927 00:47:23.471956 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:23.471982 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:23.472534 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:23.473012 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:23.474101 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.475256 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.476533 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.477221 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.477402 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.480785 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:23.480792 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:23.480804 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:23.480835 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:23.482808 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.483318 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.483323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.483330 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.483442 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.483747 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.483750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.483752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.484410 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:23.484614 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:23.484943 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 00:47:23.485344 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":170,"threads_started":1}
I20260927 00:47:23.488385 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.478023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49818 (local address 127.0.98.62:34945)
0927 00:47:23.478227 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:23.478232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:23.478601 (+ 369us) server_negotiation.cc:407] Connection header received
0927 00:47:23.478784 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:23.478787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:23.478838 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:23.478909 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:23.479859 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.480616 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.481612 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.481982 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.482108 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.484808 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:23.484811 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:23.484814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:23.484847 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:23.486415 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.486937 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.486940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.486942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.487042 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.487367 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.487371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.487373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.487635 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:23.487739 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:23.488060 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 00:47:23.488205 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 00:47:23.488387 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.477838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.0.1:49818)
0927 00:47:23.478437 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:23.478502 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:23.478659 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:23.479005 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:23.479014 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:23.479563 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:23.479716 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.479722 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.480782 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.480785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.481485 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.481494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.482109 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.482112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.482236 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.482751 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:23.482763 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:23.484610 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:23.486536 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.486543 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.486549 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.486801 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.487156 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.487159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.487161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.487255 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.487738 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:23.487744 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:23.487853 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 00:47:23.487977 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 00:47:23.488143 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":221,"threads_started":1}
I20260927 00:47:23.489740 7219 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:47:23 dist-test-slave-86rs krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470043, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 00:47:23.498770 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.490336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34013 (local address 127.0.0.1:34208)
0927 00:47:23.490465 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:23.490481 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:23.490555 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:23.491032 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:23.491035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:23.491249 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:23.491355 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.491360 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.492386 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.492393 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.492979 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.492987 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.493470 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.493474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.493575 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.494074 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:23.494085 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:23.495410 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:23.497156 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.497160 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.497162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.497359 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.497668 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.497671 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.497673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.497758 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.498162 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:23.498165 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:23.498271 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 00:47:23.498406 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 00:47:23.498563 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 00:47:23.498783 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.490494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34208 (local address 127.0.98.61:34013)
0927 00:47:23.490803 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:23.490806 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:23.490819 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:47:23.490863 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:23.490865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:23.490911 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:23.491020 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:23.491479 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.492272 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.493103 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.493353 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.493472 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.495566 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:23.495570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:23.495572 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:23.495597 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:23.497035 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.497462 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.497466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.497468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.497555 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.497855 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.497858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.497860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.498062 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:23.498174 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:23.498471 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 00:47:23.498620 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":118,"threads_started":1}
I20260927 00:47:23.499796 7219 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32877 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:47:23 dist-test-slave-86rs krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470043, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 00:47:23.510715 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.500691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32877 (local address 127.0.0.1:37596)
0927 00:47:23.500852 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:23.500870 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:23.500968 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:23.501307 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:23.501310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:23.501538 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:23.501633 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.501639 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.503927 (+ 2288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.503931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.504743 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.504756 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.505382 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.505384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.505448 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.506104 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:23.506117 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:23.507608 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:23.509297 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.509300 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.509302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.509496 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.509741 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.509743 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.509744 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.509819 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.510216 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:23.510218 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:23.510292 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 00:47:23.510374 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 00:47:23.510505 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 00:47:23.510720 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.500803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37596 (local address 127.0.98.60:32877)
0927 00:47:23.500928 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:23.500932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:23.500946 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:47:23.501099 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:23.501103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:23.501160 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:23.501252 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:23.501762 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.503752 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.504890 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.505248 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.505373 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.507756 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:23.507759 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:23.507761 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:23.507784 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:23.509193 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.509582 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.509585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.509586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.509663 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.509896 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.509898 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.509899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.510106 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:23.510214 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:23.510426 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 00:47:23.510569 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":49}
I20260927 00:47:23.514601 7219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31: Bootstrap starting.
I20260927 00:47:23.517729 7219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:23.518677 7219 log.cc:824] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:23.521428 7219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31: No bootstrap required, opened a new log
I20260927 00:47:23.525277 7219 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32877 } }
I20260927 00:47:23.525635 7219 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:23.525732 7219 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3bfc9d2ca344168be147ebe68ef5d31, State: Initialized, Role: FOLLOWER
I20260927 00:47:23.526409 7219 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32877 } }
I20260927 00:47:23.527891 7225 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32877 } } }
I20260927 00:47:23.528041 7225 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:47:23.529088 7219 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:47:23.530853 6825 heartbeater.cc:503] Master 127.0.98.62:34945 was elected leader, sending a full tablet report...
W20260927 00:47:23.535458 7236 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:47:23.535565 7236 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:47:23.538971 7065 master_runner.cc:428] Detected that this master e3bfc9d2ca344168be147ebe68ef5d31 is joining an existing cluster
I20260927 00:47:23.539055 7065 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:32877
I20260927 00:47:23.541083 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 6562
I20260927 00:47:23.550458 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41565
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34945
--webserver_port=33817
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34945,127.0.98.61:34013
--tserver_master_addrs=127.0.98.62:34945,127.0.98.61:34013,127.0.98.60:32877 with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:47:23.553378 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.542917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37600 (local address 127.0.98.60:32877)
0927 00:47:23.543046 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:23.543052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:23.543956 (+ 904us) server_negotiation.cc:407] Connection header received
0927 00:47:23.544129 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:23.544132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:23.544175 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:23.544251 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:23.544987 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.545811 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.547018 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.547459 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.547585 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.549829 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:23.549834 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:23.549837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:23.549859 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:23.551704 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.552146 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.552149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.552151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.552251 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.552506 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.552509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.552511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.552718 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:23.552786 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:23.553062 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 00:47:23.553183 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:47:23.553879 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.542903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32877 (local address 127.0.0.1:37600)
0927 00:47:23.543847 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:23.543863 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:23.544029 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:23.544435 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:23.544438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:23.544718 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:23.544868 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.544876 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.545937 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.545940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.546875 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.546885 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.547752 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.547758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.547881 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.549123 (+ 1242us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:23.549140 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:23.549688 (+ 548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:23.551820 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.551823 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.551826 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.552054 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.552343 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.552347 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.552349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.552425 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.552841 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:23.552844 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:23.552918 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 00:47:23.553546 (+ 628us) client_negotiation.cc:240] Negotiation successful
0927 00:47:23.553708 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"thread_start_us":132,"threads_started":1}
I20260927 00:47:23.558073 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.543639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34220 (local address 127.0.98.61:34013)
0927 00:47:23.543791 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:23.543797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:23.548530 (+ 4733us) server_negotiation.cc:407] Connection header received
0927 00:47:23.548733 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:23.548739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:23.548815 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:23.548877 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:23.549673 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.550492 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.552035 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.552491 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.552570 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.554075 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:23.554079 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:23.554082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:23.554114 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:23.555788 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.556375 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.556378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.556381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.556473 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.556864 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.556871 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.556875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.557164 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:23.557254 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:23.557744 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 00:47:23.557895 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:47:23.558136 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.543509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34013 (local address 127.0.0.1:34220)
0927 00:47:23.548426 (+ 4917us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:23.548443 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:23.548602 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:23.549051 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:23.549057 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:23.549350 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:23.549499 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.549509 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.550788 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.550794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.551885 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.551898 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.552756 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.552762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.552900 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.553393 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:23.553412 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:23.553903 (+ 491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:23.555906 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.555912 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.555917 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.556177 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.556580 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.556587 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.556592 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.556734 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.557338 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:23.557343 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:23.557475 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 00:47:23.557670 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 00:47:23.557855 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4834,"thread_start_us":81,"threads_started":1}
I20260927 00:47:23.573391 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.542921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49822 (local address 127.0.98.62:34945)
0927 00:47:23.543051 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:23.543055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:23.547310 (+ 4255us) server_negotiation.cc:407] Connection header received
0927 00:47:23.547517 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:23.547521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:23.547589 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:23.547661 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:23.564993 (+ 17332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.565919 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.567246 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.567635 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.567752 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.569295 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:23.569300 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:23.569303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:23.569335 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:23.571331 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.571878 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.571883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.571885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.572005 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.572359 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.572363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.572365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.572638 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:23.572740 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:23.573048 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 00:47:23.573197 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:47:23.573426 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.543142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.0.1:49822)
0927 00:47:23.547205 (+ 4063us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:23.547222 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:23.547395 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:23.547824 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:23.547827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:23.564548 (+ 16721us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:23.564762 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.564775 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.566065 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.566068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.567095 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.567106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.567773 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.567776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.567897 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.568610 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:23.568633 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:23.569125 (+ 492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:23.571470 (+ 2345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.571473 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.571476 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.571767 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.572094 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.572097 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.572099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.572193 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.572733 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:23.572736 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:23.572851 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 00:47:23.572953 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 00:47:23.573185 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3977,"thread_start_us":91,"threads_started":1}
Sep 27 00:47:23 dist-test-slave-86rs krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470041, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 00:47:23.588827 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.578374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32877 (local address 127.0.0.1:37606)
0927 00:47:23.578581 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:23.578599 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:23.578716 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:23.579032 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:23.579035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:23.579263 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:23.579397 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.579404 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.580436 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.580440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.581404 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.581417 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.582001 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.582004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.582119 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.582724 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:23.582741 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:23.584614 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:23.586829 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.586832 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.586835 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.587100 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.587545 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.587548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.587550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.587658 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.588137 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:23.588140 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:23.588281 (+ 141us) client_negotiation.cc:769] Sending connection context
0927 00:47:23.588392 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 00:47:23.588583 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260927 00:47:23.589430 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.578460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37606 (local address 127.0.98.60:32877)
0927 00:47:23.578589 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:23.578593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:23.578667 (+ 74us) server_negotiation.cc:407] Connection header received
0927 00:47:23.578851 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:23.578854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:23.578915 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:23.579001 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:23.579517 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.580323 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.581557 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.581874 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.581981 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.584848 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:23.584853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:23.584856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:23.584891 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:23.586685 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.587222 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.587227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.587231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.587437 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.587755 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.587758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.587760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.588002 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:23.588092 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:23.589127 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0927 00:47:23.589271 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:47:23.590683 6500 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:32877: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 32877
}
attrs {
promote: true
}
}
dest_uuid: "5259924bdea44542b5131a31ce919b1d"
cas_config_opid_index: 5
I20260927 00:47:23.591442 6500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: true } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: true } }
I20260927 00:47:23.592798 7061 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e3bfc9d2ca344168be147ebe68ef5d31 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: true } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: true } } }
I20260927 00:47:23.593591 7193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:23.594712 7061 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:32877
I20260927 00:47:23.594897 7246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5259924bdea44542b5131a31ce919b1d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: true } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: true } } }
I20260927 00:47:23.595012 7246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:23.595007 7193 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [term 1 FOLLOWER]: Refusing update from remote peer 5259924bdea44542b5131a31ce919b1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 00:47:23.595814 7246 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:47:23.595876 7065 master.cc:537] Master@127.0.98.60:32877 shutting down...
W20260927 00:47:23.599520 6484 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d -> Peer e3bfc9d2ca344168be147ebe68ef5d31 (127.0.98.60:32877): Couldn't send request to peer e3bfc9d2ca344168be147ebe68ef5d31. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 00:47:23.602077 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.595444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34013 (local address 127.0.0.1:34230)
0927 00:47:23.596369 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:23.596388 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:23.596493 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:23.597137 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:23.597140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:23.597162 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:23.597284 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.597290 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.598428 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.598433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.600094 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.600108 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.601514 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.601517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.601639 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.601714 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 00:47:23.601792 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 00:47:23.601897 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806}
I20260927 00:47:23.602718 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.595570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34230 (local address 127.0.98.61:34013)
0927 00:47:23.596716 (+ 1146us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:23.596722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:23.596739 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:47:23.596803 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:23.596807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:23.596882 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:23.596978 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:23.597432 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.598319 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.600291 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.601097 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.601218 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.601436 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:23.602405 (+ 969us) server_negotiation.cc:299] Negotiation successful
0927 00:47:23.602533 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1009}
I20260927 00:47:23.603338 7111 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [term 1 LEARNER]: Refusing update from remote peer 5259924bdea44542b5131a31ce919b1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 00:47:23.603896 7246 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 00:47:23.603996 7065 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:47:23.606377 7111 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e3bfc9d2ca344168be147ebe68ef5d31 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: true } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: true } } }
I20260927 00:47:23.608841 7065 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:23.609037 7065 tablet_replica.cc:330] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31: stopping tablet replica
I20260927 00:47:23.610608 7140 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5259924bdea44542b5131a31ce919b1d. Latest consensus state: current_term: 1 leader_uuid: "5259924bdea44542b5131a31ce919b1d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: true } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: true } } }
I20260927 00:47:23.610780 7140 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [sys.catalog]: This master's current role is: LEARNER
I20260927 00:47:23.610889 7140 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5259924bdea44542b5131a31ce919b1d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: true } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: true } } }
I20260927 00:47:23.610953 7140 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [sys.catalog]: This master's current role is: LEARNER
I20260927 00:47:23.613919 7246 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d: attempting to promote NON_VOTER ceb92bf11e1b41a19b0118bb8a88002d to VOTER
I20260927 00:47:23.614513 7246 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: true } }
I20260927 00:47:23.615353 7065 master.cc:552] Master@127.0.98.60:32877 shutdown complete.
I20260927 00:47:23.615446 7065 master_runner.cc:305] Clearing existing system tablet
I20260927 00:47:23.615769 7065 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:23.615859 7065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:23.616304 7110 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [term 1 LEARNER]: Refusing update from remote peer 5259924bdea44542b5131a31ce919b1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 00:47:23.616784 7062 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 00:47:23.618459 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.618077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32877 (local address 127.0.0.1:37610)
0927 00:47:23.618269 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:23.618361 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 00:47:23.618827 7062 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ceb92bf11e1b41a19b0118bb8a88002d (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: true } } }
I20260927 00:47:23.619355 7110 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ceb92bf11e1b41a19b0118bb8a88002d (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: true } } }
I20260927 00:47:23.621651 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.621321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32877 (local address 127.0.0.1:37626)
0927 00:47:23.621483 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:23.621540 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260927 00:47:23.621788 6484 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d -> Peer e3bfc9d2ca344168be147ebe68ef5d31 (127.0.98.60:32877): Couldn't send request to peer e3bfc9d2ca344168be147ebe68ef5d31. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32877: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 00:47:23.622069 7248 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5259924bdea44542b5131a31ce919b1d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: true } } }
I20260927 00:47:23.622273 7248 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:47:23.622534 7062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5259924bdea44542b5131a31ce919b1d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: true } } }
I20260927 00:47:23.622668 7062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:23.622759 7061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5259924bdea44542b5131a31ce919b1d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: true } } }
I20260927 00:47:23.622838 7061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:23.628559 7065 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.001s sys 0.000s
I20260927 00:47:23.630738 7251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:23.631007 7065 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:23.631402 7065 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 00:47:23.631459 7065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-2/wal
uuid: "e3bfc9d2ca344168be147ebe68ef5d31"
format_stamp: "Formatted at 2026-09-27 00:47:23 on dist-test-slave-86rs"
server_key: "f6548a99562ae8e1b0f02f3812bb1d83"
server_key_iv: "28597a09fa36edff90cce049404fe745"
server_key_version: "encryptionkey@0"
I20260927 00:47:23.634578 7065 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:23.638003 7065 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 00:47:23.638227 7065 log.cc:1196] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 00:47:23.638516 7065 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31: Deleting consensus metadata
I20260927 00:47:23.638697 7065 master_runner.cc:315] Copying system tablet from 127.0.98.62:34945
I20260927 00:47:23.640341 7065 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:34945
I20260927 00:47:23.667075 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.644886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49830 (local address 127.0.98.62:34945)
0927 00:47:23.646905 (+ 2019us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:23.646912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:23.656794 (+ 9882us) server_negotiation.cc:407] Connection header received
0927 00:47:23.657020 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:23.657024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:23.657102 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:23.657238 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:23.657969 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.658941 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.660370 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.660724 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.660840 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.663135 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:23.663139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:23.663142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:23.663178 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:23.664981 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.665541 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.665545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.665548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.665664 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.665966 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.665970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.665971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.666243 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:23.666321 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:23.666715 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 00:47:23.666876 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1877}
I20260927 00:47:23.667783 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.645125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.0.1:49830)
0927 00:47:23.656589 (+ 11464us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:23.656614 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:23.656870 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:23.657284 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:23.657289 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:23.657616 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:23.657803 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.657813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.659075 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.659083 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.660175 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.660189 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.661047 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.661051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.661145 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.662360 (+ 1215us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:23.662377 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:23.662927 (+ 550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:23.665131 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.665135 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.665138 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.665417 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.665778 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.665781 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.665783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.665872 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.666393 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:23.666396 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:23.666510 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 00:47:23.667389 (+ 879us) client_negotiation.cc:240] Negotiation successful
0927 00:47:23.667585 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11285,"thread_start_us":123,"threads_started":1}
I20260927 00:47:23.668373 6520 tablet_copy_service.cc:140] P 5259924bdea44542b5131a31ce919b1d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e3bfc9d2ca344168be147ebe68ef5d31 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:49830)
I20260927 00:47:23.668497 6520 tablet_copy_service.cc:161] P 5259924bdea44542b5131a31ce919b1d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e3bfc9d2ca344168be147ebe68ef5d31 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:49830: session id = e3bfc9d2ca344168be147ebe68ef5d31-00000000000000000000000000000000
I20260927 00:47:23.669922 6520 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 00:47:23.671988 7065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 00:47:23.675303 7065 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31: tablet copy: Starting download of 0 data blocks...
I20260927 00:47:23.675535 7065 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31: tablet copy: Starting download of 1 WAL segments...
I20260927 00:47:23.677757 7065 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 00:47:23.680389 6520 tablet_copy_service.cc:342] P 5259924bdea44542b5131a31ce919b1d: Request end of tablet copy session e3bfc9d2ca344168be147ebe68ef5d31-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:49830
I20260927 00:47:23.680704 6520 tablet_copy_service.cc:433] P 5259924bdea44542b5131a31ce919b1d: ending tablet copy session e3bfc9d2ca344168be147ebe68ef5d31-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e3bfc9d2ca344168be147ebe68ef5d31
I20260927 00:47:23.692413 7065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:23.694773 7258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:23.702091 7261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:23.702401 7065 server_base.cc:1034] running on GCE node
W20260927 00:47:23.702540 7259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:23.702829 7065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:23.703287 7065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:23.704444 7065 hybrid_clock.cc:648] HybridClock initialized: now 1790470043704421 us; error 45 us; skew 500 ppm
Sep 27 00:47:23 dist-test-slave-86rs krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470043, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:23.706753 7065 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 00:47:23.707556 7065 webserver.cc:460] Webserver started at http://127.0.98.60:33543/ using document root <none> and password file <none>
I20260927 00:47:23.707825 7065 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:23.707876 7065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:23.709795 7065 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 00:47:23.711150 7268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:23.711488 7065 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 00:47:23.711562 7065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-2/wal
uuid: "e3bfc9d2ca344168be147ebe68ef5d31"
format_stamp: "Formatted at 2026-09-27 00:47:23 on dist-test-slave-86rs"
server_key: "f6548a99562ae8e1b0f02f3812bb1d83"
server_key_iv: "28597a09fa36edff90cce049404fe745"
server_key_version: "encryptionkey@0"
I20260927 00:47:23.711674 7065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:23.727182 7065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:23.730202 7065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:23.736276 7065 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:32877
I20260927 00:47:23.737190 7065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/master-2/data/info.pb
I20260927 00:47:23.739372 7321 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:47:23.740733 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31: Bootstrap starting.
I20260927 00:47:23.744549 7320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:32877 every 8 connection(s)
I20260927 00:47:23.753943 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:47:23.754333 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31: Bootstrap complete.
I20260927 00:47:23.755066 7321 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: true } }
I20260927 00:47:23.755216 7321 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e3bfc9d2ca344168be147ebe68ef5d31, State: Initialized, Role: LEARNER
I20260927 00:47:23.755306 7321 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: true } }
I20260927 00:47:23.755661 7327 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: true } } }
I20260927 00:47:23.755785 7327 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [sys.catalog]: This master's current role is: LEARNER
I20260927 00:47:23.755857 7321 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:47:23.760293 7338 catalog_manager.cc:1259] Loaded cluster ID: 6864942181e149ff81377ee6995feab8
I20260927 00:47:23.760383 7338 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31: loading cluster ID for follower catalog manager: success
W20260927 00:47:23.761585 7245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:23.762025 7245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:23.762099 7245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 00:47:23.762933 7338 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31: acquiring CA information for follower catalog manager: success
I20260927 00:47:23.764695 7338 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 00:47:23.767537 7245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:23.767632 7245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:23.767742 7245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:47:23.774577 7245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33817
--tserver_master_addrs=127.0.98.62:34945,127.0.98.61:34013,127.0.98.60:32877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:23.776187 7245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:23.777901 7245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:23.790915 7340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:23.790915 7343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:23.791082 7245 server_base.cc:1034] running on GCE node
W20260927 00:47:23.790915 7341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:23.791813 7245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:23.792750 7245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:23.793977 7245 hybrid_clock.cc:648] HybridClock initialized: now 1790470043793924 us; error 65 us; skew 500 ppm
I20260927 00:47:23.795565 6959 heartbeater.cc:503] Master 127.0.98.62:34945 was elected leader, sending a full tablet report...
Sep 27 00:47:23 dist-test-slave-86rs krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470043, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:23.797776 7245 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 00:47:23.799005 7245 webserver.cc:460] Webserver started at http://127.0.98.1:33817/ using document root <none> and password file <none>
I20260927 00:47:23.799786 7245 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:23.799921 7245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:23.805368 7245 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:47:23.808151 7350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:23.809875 7245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:47:23.810057 7245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "8ed31e81e1864605b6c77a8123fdb062"
format_stamp: "Formatted at 2026-09-27 00:47:22 on dist-test-slave-86rs"
server_key: "eb0238b16014acf6238db01a93e1ffc1"
server_key_iv: "e1abd7e76eee7e9995ce14d6bf364cab"
server_key_version: "encryptionkey@0"
I20260927 00:47:23.810859 7245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:23.830966 7245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:23.835606 7245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:23.835881 7245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:23.836803 7245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:23.838069 7245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:23.838150 7245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:23.838238 7245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:23.838279 7245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:23.850435 7245 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41565
I20260927 00:47:23.850452 7463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41565 every 8 connection(s)
I20260927 00:47:23.851831 7245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:47:23.853884 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 7245
I20260927 00:47:23.854180 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 6696
I20260927 00:47:23.865028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34123
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34945
--webserver_port=44893
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34945,127.0.98.61:34013
--tserver_master_addrs=127.0.98.62:34945,127.0.98.61:34013,127.0.98.60:32877 with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:47:23 dist-test-slave-86rs krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470043, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 00:47:23 dist-test-slave-86rs krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470043, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 00:47:23 dist-test-slave-86rs krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470043, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 00:47:23.873308 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.857397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55127 (local address 127.0.98.61:34013)
0927 00:47:23.857585 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:23.857592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:23.861155 (+ 3563us) server_negotiation.cc:407] Connection header received
0927 00:47:23.861345 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:23.861349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:23.861430 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:23.861490 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:23.862080 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.862858 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.864058 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.864431 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.864509 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.868335 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:23.868341 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:23.868345 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:23.868382 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:23.870705 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.871204 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.871208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.871210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.871315 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.871600 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.871605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.871607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.871881 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:23.871959 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:23.872941 (+ 982us) server_negotiation.cc:299] Negotiation successful
0927 00:47:23.873098 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 00:47:23.873939 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.856658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33741 (local address 127.0.98.60:32877)
0927 00:47:23.856946 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:23.856954 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:23.856977 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:47:23.857792 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:23.857800 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:23.857882 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:23.857995 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:23.859584 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.860880 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.863019 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.863470 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.863598 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.867732 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:23.867739 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:23.867743 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:23.867791 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:23.870095 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.870994 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.870999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.871002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.871098 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.871640 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.871644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.871648 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.871881 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:23.871972 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:23.873233 (+ 1261us) server_negotiation.cc:299] Negotiation successful
0927 00:47:23.873758 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":100,"threads_started":1}
I20260927 00:47:23.874537 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.854366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38069 (local address 127.0.98.62:34945)
0927 00:47:23.854949 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:23.854955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:23.860794 (+ 5839us) server_negotiation.cc:407] Connection header received
0927 00:47:23.860961 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:23.860964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:23.861037 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:23.861131 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:23.861856 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.862717 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.864059 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.864431 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.864509 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.868202 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:23.868210 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:23.868214 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:23.868279 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:23.870198 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.871034 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.871041 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.871044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.871156 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:23.871586 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:23.871591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:23.871595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:23.871926 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:23.872034 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:23.874119 (+ 2085us) server_negotiation.cc:299] Negotiation successful
0927 00:47:23.874288 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446}
I20260927 00:47:23.875214 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.854891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.98.1:38069)
0927 00:47:23.860670 (+ 5779us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:23.860684 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:23.860837 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:23.861266 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:23.861270 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:23.861532 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:23.861710 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.861717 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.862859 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.862862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.863924 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.863931 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.864686 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.864689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.864808 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.865589 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:23.865610 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:23.868013 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:23.870340 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.870346 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.870352 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.870896 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.871278 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.871284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.871289 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.871463 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.872139 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:23.872144 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:23.872597 (+ 453us) client_negotiation.cc:769] Sending connection context
0927 00:47:23.872881 (+ 284us) client_negotiation.cc:240] Negotiation successful
0927 00:47:23.873414 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5670,"mutex_wait_us":1817,"spinlock_wait_cycles":49408,"thread_start_us":117,"threads_started":1}
I20260927 00:47:23.875384 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.854891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32877 (local address 127.0.98.1:33741)
0927 00:47:23.855978 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:23.856046 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:23.857453 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:23.858221 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:23.858231 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:23.858719 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:23.859386 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.859402 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.861061 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.861067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.862864 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.862878 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.863820 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.863826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.864099 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.864972 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:23.864995 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:23.867475 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:23.870305 (+ 2830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.870313 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.870341 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.870870 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.871213 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.871217 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.871219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.871451 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.872033 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:23.872041 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:23.872595 (+ 554us) client_negotiation.cc:769] Sending connection context
0927 00:47:23.872880 (+ 285us) client_negotiation.cc:240] Negotiation successful
0927 00:47:23.875073 (+ 2193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"mutex_wait_us":1882,"thread_start_us":621,"threads_started":1}
I20260927 00:47:23.876222 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:23.857300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34013 (local address 127.0.98.1:55127)
0927 00:47:23.861049 (+ 3749us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:23.861063 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:23.861226 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:23.861645 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:23.861648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:23.861842 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:23.861952 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.861958 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.862985 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.862988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.863921 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:23.863932 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:23.864682 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:23.864685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:23.864836 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:23.865589 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:23.865609 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:23.868145 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:23.870841 (+ 2696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.870844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.870846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.871085 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.871412 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:23.871415 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:23.871417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:23.871501 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:23.872033 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:23.872041 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:23.872595 (+ 554us) client_negotiation.cc:769] Sending connection context
0927 00:47:23.873591 (+ 996us) client_negotiation.cc:240] Negotiation successful
0927 00:47:23.873757 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3689,"mutex_wait_us":106,"spinlock_wait_cycles":61312,"thread_start_us":94,"threads_started":1}
I20260927 00:47:23.877600 7465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34945
I20260927 00:47:23.877723 7466 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34013
I20260927 00:47:23.877759 7464 heartbeater.cc:347] Connected to a master server at 127.0.98.60:32877
I20260927 00:47:23.877929 7466 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:23.877930 7465 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:23.877929 7464 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:23.878762 7464 heartbeater.cc:511] Master 127.0.98.60:32877 requested a full tablet report, sending...
I20260927 00:47:23.878783 7465 heartbeater.cc:511] Master 127.0.98.62:34945 requested a full tablet report, sending...
I20260927 00:47:23.878779 7466 heartbeater.cc:511] Master 127.0.98.61:34013 requested a full tablet report, sending...
I20260927 00:47:23.880540 6500 ts_manager.cc:194] Re-registered known tserver with Master: 8ed31e81e1864605b6c77a8123fdb062 (127.0.98.1:41565)
I20260927 00:47:23.881078 7285 ts_manager.cc:194] Registered new tserver with Master: 8ed31e81e1864605b6c77a8123fdb062 (127.0.98.1:41565)
I20260927 00:47:23.881302 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38069
I20260927 00:47:23.882287 7101 ts_manager.cc:194] Registered new tserver with Master: 8ed31e81e1864605b6c77a8123fdb062 (127.0.98.1:41565)
W20260927 00:47:24.015632 7472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:24.016037 7472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:24.016090 7472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:24.021029 7472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:24.021150 7472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:24.021226 7472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:47:24.027786 7472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34123
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44893
--tserver_master_addrs=127.0.98.62:34945,127.0.98.61:34013,127.0.98.60:32877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:24.029666 7472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:24.031574 7472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:24.044893 7478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:24.044960 7477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:24.044965 7480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:24.045377 7472 server_base.cc:1034] running on GCE node
I20260927 00:47:24.046017 7472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:24.046861 7472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:24.048045 7472 hybrid_clock.cc:648] HybridClock initialized: now 1790470044048026 us; error 41 us; skew 500 ppm
Sep 27 00:47:24 dist-test-slave-86rs krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470044, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:24.051648 7472 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 00:47:24.053007 7472 webserver.cc:460] Webserver started at http://127.0.98.2:44893/ using document root <none> and password file <none>
I20260927 00:47:24.053798 7472 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:24.053929 7472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:24.059572 7472 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:47:24.062726 7487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:24.064705 7472 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:47:24.064898 7472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86"
format_stamp: "Formatted at 2026-09-27 00:47:22 on dist-test-slave-86rs"
server_key: "dc4c825a77bd767bf6f7f95199b42c91"
server_key_iv: "3c4f52435c646bc0ff3252dac0488e37"
server_key_version: "encryptionkey@0"
I20260927 00:47:24.065689 7472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:24.078441 7472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:24.083005 7472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:24.083300 7472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:24.084182 7472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:24.085626 7472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:24.085711 7472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:24.085798 7472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:24.085848 7472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:24.097455 7472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34123
I20260927 00:47:24.097517 7600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34123 every 8 connection(s)
I20260927 00:47:24.098924 7472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:47:24.105417 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 7472
I20260927 00:47:24.105701 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 6830
I20260927 00:47:24.114873 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42375
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34945
--webserver_port=44215
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34945,127.0.98.61:34013
--tserver_master_addrs=127.0.98.62:34945,127.0.98.61:34013,127.0.98.60:32877 with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:47:24 dist-test-slave-86rs krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470044, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 00:47:24 dist-test-slave-86rs krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470044, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 00:47:24 dist-test-slave-86rs krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470044, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:47:24.127561 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.102099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52325 (local address 127.0.98.62:34945)
0927 00:47:24.102306 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:24.102312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:24.104146 (+ 1834us) server_negotiation.cc:407] Connection header received
0927 00:47:24.108518 (+ 4372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:24.108523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:24.108615 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:24.108727 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:24.111706 (+ 2979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.112786 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.115338 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.115751 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.115855 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.122031 (+ 6176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:24.122040 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:24.122045 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:24.122092 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:24.124680 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:24.125448 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:24.125453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:24.125455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:24.125792 (+ 337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:24.126167 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:24.126172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:24.126173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:24.126455 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:24.126541 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:24.127114 (+ 573us) server_negotiation.cc:299] Negotiation successful
0927 00:47:24.127307 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:47:24.127604 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.102273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35389 (local address 127.0.98.60:32877)
0927 00:47:24.103194 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:24.103200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:24.104905 (+ 1705us) server_negotiation.cc:407] Connection header received
0927 00:47:24.108933 (+ 4028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:24.108938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:24.109010 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:24.109101 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:24.111702 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.112773 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.115436 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.115877 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.115968 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.121199 (+ 5231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:24.121208 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:24.121214 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:24.121271 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:24.123759 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:24.124817 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:24.124824 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:24.124828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:24.124953 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:24.125548 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:24.125554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:24.125559 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:24.125915 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:24.126028 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:24.127117 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0927 00:47:24.127332 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777}
I20260927 00:47:24.128397 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.104923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44899 (local address 127.0.98.61:34013)
0927 00:47:24.105069 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:24.105075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:24.109070 (+ 3995us) server_negotiation.cc:407] Connection header received
0927 00:47:24.109323 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:24.109329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:24.109410 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:24.109495 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:24.112092 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.113044 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.115582 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.116016 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.116133 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.121699 (+ 5566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:24.121706 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:24.121709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:24.121754 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:24.124657 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:24.125456 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:24.125460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:24.125462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:24.125588 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:24.126040 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:24.126046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:24.126051 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:24.126348 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:24.126449 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:24.127988 (+ 1539us) server_negotiation.cc:299] Negotiation successful
0927 00:47:24.128171 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:47:24.129808 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.102966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32877 (local address 127.0.98.2:35389)
0927 00:47:24.104754 (+ 1788us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:24.104776 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:24.108006 (+ 3230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:24.109247 (+ 1241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:24.109250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:24.109973 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:24.111433 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:24.111463 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.113027 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.113031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:24.115276 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:24.115289 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.116284 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.116287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:24.116733 (+ 446us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.118021 (+ 1288us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:24.118040 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:24.120754 (+ 2714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:24.123970 (+ 3216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:24.123983 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:24.124017 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:24.124643 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:24.125103 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:24.125108 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:24.125113 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:24.125408 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:24.126127 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:24.126140 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:24.126716 (+ 576us) client_negotiation.cc:769] Sending connection context
0927 00:47:24.127812 (+ 1096us) client_negotiation.cc:240] Negotiation successful
0927 00:47:24.128038 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1644,"mutex_wait_us":634,"spinlock_wait_cycles":126848,"thread_start_us":265,"threads_started":1}
I20260927 00:47:24.129808 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.102966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.98.2:52325)
0927 00:47:24.103866 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:24.103954 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:24.108037 (+ 4083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:24.109068 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:24.109088 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:24.109992 (+ 904us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:24.111433 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:24.111463 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.113027 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.113031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:24.115136 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:24.115155 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.116178 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.116186 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:24.116697 (+ 511us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.118121 (+ 1424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:24.118142 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:24.121775 (+ 3633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:24.124836 (+ 3061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:24.124840 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:24.124842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:24.125289 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:24.125925 (+ 636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:24.125928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:24.125930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:24.126024 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:24.126619 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:24.126622 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:24.126713 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 00:47:24.128704 (+ 1991us) client_negotiation.cc:240] Negotiation successful
0927 00:47:24.128899 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":496,"spinlock_wait_cycles":42368,"thread_start_us":262,"threads_started":1}
I20260927 00:47:24.130740 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.104731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34013 (local address 127.0.98.2:44899)
0927 00:47:24.108932 (+ 4201us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:24.108953 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:24.109181 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:24.109695 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:24.109701 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:24.110694 (+ 993us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:24.111885 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:24.111902 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.113289 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.113296 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:24.115318 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:24.115334 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.116416 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.116423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:24.116728 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.117859 (+ 1131us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:24.117893 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:24.121481 (+ 3588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:24.124836 (+ 3355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:24.124840 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:24.124842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:24.125289 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:24.125720 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:24.125724 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:24.125726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:24.125832 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:24.126515 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:24.126519 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:24.126713 (+ 194us) client_negotiation.cc:769] Sending connection context
0927 00:47:24.127030 (+ 317us) client_negotiation.cc:240] Negotiation successful
0927 00:47:24.127586 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4064,"mutex_wait_us":979,"spinlock_wait_cycles":11136,"thread_start_us":149,"threads_started":1}
I20260927 00:47:24.131891 7601 heartbeater.cc:347] Connected to a master server at 127.0.98.60:32877
I20260927 00:47:24.132001 7603 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34013
I20260927 00:47:24.132206 7601 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:24.132205 7603 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:24.132320 7602 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34945
I20260927 00:47:24.132448 7602 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:24.133121 7602 heartbeater.cc:511] Master 127.0.98.62:34945 requested a full tablet report, sending...
I20260927 00:47:24.133121 7601 heartbeater.cc:511] Master 127.0.98.60:32877 requested a full tablet report, sending...
I20260927 00:47:24.133857 7603 heartbeater.cc:511] Master 127.0.98.61:34013 requested a full tablet report, sending...
I20260927 00:47:24.134804 6500 ts_manager.cc:194] Re-registered known tserver with Master: 1fedc53da95e4fc5bcdc6e2a3ee28a86 (127.0.98.2:34123)
I20260927 00:47:24.134872 7101 ts_manager.cc:194] Registered new tserver with Master: 1fedc53da95e4fc5bcdc6e2a3ee28a86 (127.0.98.2:34123)
I20260927 00:47:24.135042 7285 ts_manager.cc:194] Registered new tserver with Master: 1fedc53da95e4fc5bcdc6e2a3ee28a86 (127.0.98.2:34123)
I20260927 00:47:24.135730 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52325
I20260927 00:47:24.156611 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.151550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37638 (local address 127.0.98.60:32877)
0927 00:47:24.151739 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:24.151744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:24.152007 (+ 263us) server_negotiation.cc:407] Connection header received
0927 00:47:24.152177 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:24.152181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:24.152290 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:24.152415 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:24.152813 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.153638 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.154895 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.155662 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.155789 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.155993 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:24.156342 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 00:47:24.156443 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 00:47:24.156669 7610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.151420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32877 (local address 127.0.0.1:37638)
0927 00:47:24.151887 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:24.151904 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:24.152053 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:24.152493 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:24.152497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:24.152538 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:24.152670 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:24.152676 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.153810 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.153814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:24.154727 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:24.154741 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.155841 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.155845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:24.156103 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.156164 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 00:47:24.156319 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 00:47:24.156444 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":145,"threads_started":1}
I20260927 00:47:24.157531 7295 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 00:47:24.158056 7327 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5259924bdea44542b5131a31ce919b1d. Latest consensus state: current_term: 1 leader_uuid: "5259924bdea44542b5131a31ce919b1d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: true } } }
I20260927 00:47:24.158284 7327 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [sys.catalog]: This master's current role is: LEARNER
W20260927 00:47:24.273289 7608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:24.273705 7608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:24.273761 7608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:24.278952 7608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:24.279076 7608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:24.279158 7608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:47:24.285898 7608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44215
--tserver_master_addrs=127.0.98.62:34945,127.0.98.61:34013,127.0.98.60:32877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:24.287410 7608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:24.288795 7608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:24.301908 7616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:24.301883 7618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:24.301885 7615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:24.302363 7608 server_base.cc:1034] running on GCE node
I20260927 00:47:24.302935 7608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:24.303836 7608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:24.305034 7608 hybrid_clock.cc:648] HybridClock initialized: now 1790470044304987 us; error 72 us; skew 500 ppm
Sep 27 00:47:24 dist-test-slave-86rs krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470044, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:24.308580 7608 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 00:47:24.310295 7608 webserver.cc:460] Webserver started at http://127.0.98.3:44215/ using document root <none> and password file <none>
I20260927 00:47:24.311275 7608 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:24.311419 7608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:24.317806 7608 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:47:24.321317 7625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:24.323467 7608 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260927 00:47:24.323655 7608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "f4312a0972ce4ab3905b603a432eacb8"
format_stamp: "Formatted at 2026-09-27 00:47:22 on dist-test-slave-86rs"
server_key: "e03f7bdb9ae77d332406ca54b3fe0974"
server_key_iv: "b81f599661f83f20d1de4117e1454b03"
server_key_version: "encryptionkey@0"
I20260927 00:47:24.324695 7608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:24.347892 7608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:24.352406 7608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:24.352618 7608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:24.353379 7608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:24.354544 7608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:24.354601 7608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:24.354651 7608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:24.354668 7608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:24.367765 7608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42375
I20260927 00:47:24.367789 7738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42375 every 8 connection(s)
I20260927 00:47:24.369359 7608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:47:24.376379 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 7608
Sep 27 00:47:24 dist-test-slave-86rs krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470044, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 00:47:24 dist-test-slave-86rs krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470044, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 00:47:24 dist-test-slave-86rs krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470044, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 00:47:24.391800 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.374336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35143 (local address 127.0.98.61:34013)
0927 00:47:24.374498 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:24.374502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:24.374593 (+ 91us) server_negotiation.cc:407] Connection header received
0927 00:47:24.375257 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:24.375261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:24.375325 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:24.375396 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:24.377423 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.378222 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.380855 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.381198 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.381309 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.386822 (+ 5513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:24.386826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:24.386829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:24.386860 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:24.388760 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:24.389600 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:24.389606 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:24.389610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:24.389716 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:24.390189 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:24.390193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:24.390195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:24.390456 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:24.390552 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:24.391421 (+ 869us) server_negotiation.cc:299] Negotiation successful
0927 00:47:24.391592 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 00:47:24.394533 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.372203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44863 (local address 127.0.98.60:32877)
0927 00:47:24.372438 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:24.372443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:24.374249 (+ 1806us) server_negotiation.cc:407] Connection header received
0927 00:47:24.375258 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:24.375261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:24.375354 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:24.375434 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:24.377378 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.378279 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.380443 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.380740 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.380861 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.389948 (+ 9087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:24.389952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:24.389956 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:24.389992 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:24.392273 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:24.392920 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:24.392924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:24.392926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:24.393031 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:24.393389 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:24.393393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:24.393396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:24.393674 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:24.393760 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:24.394161 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 00:47:24.394310 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260927 00:47:24.394857 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.372013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53183 (local address 127.0.98.62:34945)
0927 00:47:24.372203 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:24.372208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:24.373674 (+ 1466us) server_negotiation.cc:407] Connection header received
0927 00:47:24.375162 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:24.375166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:24.375245 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:24.375341 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:24.377416 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.378280 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.380568 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.380956 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.381099 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.386992 (+ 5893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:24.386996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:24.386999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:24.387031 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:24.389769 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:24.392922 (+ 3153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:24.392926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:24.392928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:24.393040 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:24.393742 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:24.393746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:24.393747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:24.394002 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:24.394100 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:24.394502 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 00:47:24.394654 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 00:47:24.395453 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.372728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.98.3:53183)
0927 00:47:24.373454 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:24.373526 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:24.374840 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:24.375605 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:24.375614 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:24.376391 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:24.377270 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:24.377279 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.378419 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.378422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:24.380442 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:24.380449 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.381138 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.381143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:24.381370 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.382517 (+ 1147us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:24.382551 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:24.386861 (+ 4310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:24.392409 (+ 5548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:24.392424 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:24.392426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:24.392792 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:24.393154 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:24.393157 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:24.393159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:24.393619 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:24.394171 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:24.394173 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:24.394290 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 00:47:24.395081 (+ 791us) client_negotiation.cc:240] Negotiation successful
0927 00:47:24.395220 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":668,"spinlock_wait_cycles":8704,"thread_start_us":824,"threads_started":1}
I20260927 00:47:24.395458 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.374207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34013 (local address 127.0.98.3:35143)
0927 00:47:24.374492 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:24.374505 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:24.374876 (+ 371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:24.375605 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:24.375613 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:24.376474 (+ 861us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:24.377180 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:24.377205 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.378416 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.378419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:24.380730 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:24.380738 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.381326 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.381330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:24.381455 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.382493 (+ 1038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:24.382521 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:24.386539 (+ 4018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:24.388984 (+ 2445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:24.388992 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:24.389017 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:24.389462 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:24.389841 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:24.389844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:24.389846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:24.390078 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:24.390629 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:24.390638 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:24.391092 (+ 454us) client_negotiation.cc:769] Sending connection context
0927 00:47:24.392490 (+ 1398us) client_negotiation.cc:240] Negotiation successful
0927 00:47:24.393043 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":1181,"spinlock_wait_cycles":896,"thread_start_us":93,"threads_started":1}
I20260927 00:47:24.395798 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.372728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32877 (local address 127.0.98.3:44863)
0927 00:47:24.374144 (+ 1416us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:24.374159 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:24.374840 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:24.375605 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:24.375614 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:24.376358 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:24.377196 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:24.377207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.378417 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.378421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:24.380299 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:24.380312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.381448 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.381451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:24.381549 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.383017 (+ 1468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:24.385531 (+ 2514us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:24.388960 (+ 3429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:24.392385 (+ 3425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:24.392390 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:24.392392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:24.392792 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:24.393154 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:24.393157 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:24.393159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:24.393274 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:24.393819 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:24.393822 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:24.393925 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 00:47:24.394790 (+ 865us) client_negotiation.cc:240] Negotiation successful
0927 00:47:24.394972 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1320,"mutex_wait_us":263,"spinlock_wait_cycles":59648,"thread_start_us":217,"threads_started":1}
I20260927 00:47:24.397562 7741 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34013
I20260927 00:47:24.397912 7741 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:24.398748 7741 heartbeater.cc:511] Master 127.0.98.61:34013 requested a full tablet report, sending...
I20260927 00:47:24.399471 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.379439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34246 (local address 127.0.98.61:34013)
0927 00:47:24.382173 (+ 2734us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:24.382178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:24.382192 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:47:24.382229 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:24.382233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:24.382290 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:24.383409 (+ 1119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:24.383569 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.388306 (+ 4737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.391566 (+ 3260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.391911 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.393261 (+ 1350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.394793 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:24.394797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:24.394800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:24.394828 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:24.397041 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:24.397563 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:24.397568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:24.397569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:24.397664 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:24.398100 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:24.398103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:24.398105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:24.398346 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:24.399062 (+ 716us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:24.399138 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 00:47:24.399278 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2638,"thread_start_us":91,"threads_started":1}
I20260927 00:47:24.399471 7739 heartbeater.cc:347] Connected to a master server at 127.0.98.60:32877
I20260927 00:47:24.399569 7739 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:24.399767 7739 heartbeater.cc:511] Master 127.0.98.60:32877 requested a full tablet report, sending...
I20260927 00:47:24.397593 7740 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34945
I20260927 00:47:24.399967 7740 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:24.400025 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.378994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49832 (local address 127.0.98.62:34945)
0927 00:47:24.381728 (+ 2734us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:24.381733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:24.381748 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:47:24.381786 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:24.381789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:24.381834 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:24.381934 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:24.384582 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.385497 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.386809 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.387162 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.388876 (+ 1714us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.390194 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:24.390198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:24.390201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:24.390228 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:24.397826 (+ 7598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:24.398330 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:24.398335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:24.398337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:24.398440 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:24.398741 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:24.398746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:24.398748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:24.399177 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:24.399315 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:24.399561 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 00:47:24.399835 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2634,"thread_start_us":110,"threads_started":1}
I20260927 00:47:24.400103 7740 heartbeater.cc:511] Master 127.0.98.62:34945 requested a full tablet report, sending...
I20260927 00:47:24.401105 7285 ts_manager.cc:194] Registered new tserver with Master: f4312a0972ce4ab3905b603a432eacb8 (127.0.98.3:42375)
I20260927 00:47:24.401130 6499 ts_manager.cc:194] Re-registered known tserver with Master: f4312a0972ce4ab3905b603a432eacb8 (127.0.98.3:42375)
I20260927 00:47:24.401633 7101 ts_manager.cc:194] Registered new tserver with Master: f4312a0972ce4ab3905b603a432eacb8 (127.0.98.3:42375)
I20260927 00:47:24.401962 6499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53183
I20260927 00:47:24.405071 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.379975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37642 (local address 127.0.98.60:32877)
0927 00:47:24.380275 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:24.380284 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:24.381729 (+ 1445us) server_negotiation.cc:407] Connection header received
0927 00:47:24.381775 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:24.381778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:24.381833 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:24.381930 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:24.386181 (+ 4251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.386970 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.390199 (+ 3229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.390631 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.390751 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.396033 (+ 5282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:24.396039 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:24.396042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:24.396068 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:24.402868 (+ 6800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:24.403387 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:24.403391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:24.403393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:24.403490 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:24.403789 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:24.403792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:24.403794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:24.404025 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:24.404150 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:24.404804 (+ 654us) server_negotiation.cc:299] Negotiation successful
0927 00:47:24.404925 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":84,"threads_started":1}
I20260927 00:47:24.403599 6499 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:47:24.407971 6499 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:47:24.411372 7609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5259924bdea44542b5131a31ce919b1d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: true } } }
I20260927 00:47:24.411556 7609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:24.416222 7609 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d: attempting to promote NON_VOTER e3bfc9d2ca344168be147ebe68ef5d31 to VOTER
I20260927 00:47:24.416806 7609 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: false } }
I20260927 00:47:24.418085 7295 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [term 1 LEARNER]: Refusing update from remote peer 5259924bdea44542b5131a31ce919b1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 00:47:24.420508 7111 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [term 1 FOLLOWER]: Refusing update from remote peer 5259924bdea44542b5131a31ce919b1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 00:47:24.420610 7757 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 00:47:24.420980 7758 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 00:47:24.423066 7757 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, e3bfc9d2ca344168be147ebe68ef5d31 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: false } } }
I20260927 00:47:24.423816 7111 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, e3bfc9d2ca344168be147ebe68ef5d31 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: false } } }
I20260927 00:47:24.423789 7295 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, e3bfc9d2ca344168be147ebe68ef5d31 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: false } } }
I20260927 00:47:24.425110 7759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5259924bdea44542b5131a31ce919b1d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: false } } }
I20260927 00:47:24.425149 7757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5259924bdea44542b5131a31ce919b1d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: false } } }
I20260927 00:47:24.425243 7759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:24.425259 7757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:24.425390 7609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5259924bdea44542b5131a31ce919b1d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: false } } }
I20260927 00:47:24.425495 7609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:24.427287 7327 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5259924bdea44542b5131a31ce919b1d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: false } } }
I20260927 00:47:24.427464 7327 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:47:24.427412 7767 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5259924bdea44542b5131a31ce919b1d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: false } } }
I20260927 00:47:24.430781 7767 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:47:24.437325 7610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.430060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42375 (local address 127.0.0.1:56478)
0927 00:47:24.430304 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:24.430325 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:24.430422 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:24.432306 (+ 1884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:24.432310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:24.432333 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:24.432450 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:24.432456 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.433995 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.433999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:24.435429 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:24.435438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.436813 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.436817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:24.436913 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.436944 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 00:47:24.437024 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 00:47:24.437134 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260927 00:47:24.437467 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.430655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56478 (local address 127.0.98.3:42375)
0927 00:47:24.431667 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:24.431678 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:24.431703 (+ 25us) server_negotiation.cc:407] Connection header received
0927 00:47:24.431793 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:24.431798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:24.432098 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:24.432361 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:24.432575 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.433834 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.435573 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.436610 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.436812 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.437060 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:24.437128 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 00:47:24.437250 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":904,"thread_start_us":129,"threads_started":1}
I20260927 00:47:24.438422 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.430744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41565 (local address 127.0.0.1:40746)
0927 00:47:24.432815 (+ 2071us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:24.432832 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:24.432946 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:24.433681 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:24.433684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:24.433703 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:24.433814 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:24.433820 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.435369 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.435373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:24.436440 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:24.436450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.437929 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.437934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:24.438035 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.438066 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 00:47:24.438164 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 00:47:24.438250 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2007,"mutex_wait_us":51,"thread_start_us":93,"threads_started":1}
I20260927 00:47:24.438539 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.431262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40746 (local address 127.0.98.1:41565)
0927 00:47:24.431605 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:24.431614 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:24.432927 (+ 1313us) server_negotiation.cc:407] Connection header received
0927 00:47:24.433086 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:24.433093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:24.433523 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:24.433701 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:24.433918 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.435217 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.436813 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.437747 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.437921 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.438088 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:24.438201 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 00:47:24.438349 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":94,"threads_started":1}
I20260927 00:47:24.442140 7398 tablet_service.cc:1467] Processing CreateTablet for tablet eeaf1897e87d4b348e362db9a813e552 (DEFAULT_TABLE table=test-table [id=a3c57b1ad40a48b484cc50a1a6b08f71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:24.442147 7673 tablet_service.cc:1467] Processing CreateTablet for tablet eeaf1897e87d4b348e362db9a813e552 (DEFAULT_TABLE table=test-table [id=a3c57b1ad40a48b484cc50a1a6b08f71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:24.443661 7673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeaf1897e87d4b348e362db9a813e552. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:24.443661 7398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeaf1897e87d4b348e362db9a813e552. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:24.448987 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.430306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34123 (local address 127.0.0.1:40108)
0927 00:47:24.432412 (+ 2106us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:24.432440 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:24.433066 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:24.433843 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:24.433846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:24.433866 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:24.434003 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:24.434011 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.435047 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.435051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:24.436484 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:24.436496 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.448373 (+ 11877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.448380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:24.448522 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.448555 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 00:47:24.448670 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 00:47:24.448795 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2036,"mutex_wait_us":53,"thread_start_us":193,"threads_started":1}
I20260927 00:47:24.449095 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.430530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40108 (local address 127.0.98.2:34123)
0927 00:47:24.431153 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:24.431164 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:24.432563 (+ 1399us) server_negotiation.cc:407] Connection header received
0927 00:47:24.433237 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:24.433244 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:24.433490 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:24.433641 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:24.434108 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.434896 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.436693 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.437719 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.437869 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.438056 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:24.448755 (+ 10699us) server_negotiation.cc:299] Negotiation successful
0927 00:47:24.448919 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":142,"threads_started":1}
I20260927 00:47:24.453047 7535 tablet_service.cc:1467] Processing CreateTablet for tablet eeaf1897e87d4b348e362db9a813e552 (DEFAULT_TABLE table=test-table [id=a3c57b1ad40a48b484cc50a1a6b08f71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:24.454437 7773 tablet_bootstrap.cc:492] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062: Bootstrap starting.
I20260927 00:47:24.454491 7535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeaf1897e87d4b348e362db9a813e552. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:24.455510 7774 tablet_bootstrap.cc:492] T eeaf1897e87d4b348e362db9a813e552 P f4312a0972ce4ab3905b603a432eacb8: Bootstrap starting.
I20260927 00:47:24.457741 7773 tablet_bootstrap.cc:654] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:24.458325 7774 tablet_bootstrap.cc:654] T eeaf1897e87d4b348e362db9a813e552 P f4312a0972ce4ab3905b603a432eacb8: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:24.458931 7773 log.cc:824] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:24.459283 7774 log.cc:824] T eeaf1897e87d4b348e362db9a813e552 P f4312a0972ce4ab3905b603a432eacb8: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:24.461539 7773 tablet_bootstrap.cc:492] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062: No bootstrap required, opened a new log
I20260927 00:47:24.461648 7774 tablet_bootstrap.cc:492] T eeaf1897e87d4b348e362db9a813e552 P f4312a0972ce4ab3905b603a432eacb8: No bootstrap required, opened a new log
I20260927 00:47:24.461845 7773 ts_tablet_manager.cc:1397] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 00:47:24.461871 7774 ts_tablet_manager.cc:1397] T eeaf1897e87d4b348e362db9a813e552 P f4312a0972ce4ab3905b603a432eacb8: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 00:47:24.465543 7773 raft_consensus.cc:348] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4312a0972ce4ab3905b603a432eacb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } } peers { permanent_uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34123 } } peers { permanent_uuid: "8ed31e81e1864605b6c77a8123fdb062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41565 } }
I20260927 00:47:24.465914 7773 raft_consensus.cc:374] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:24.465997 7773 raft_consensus.cc:729] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ed31e81e1864605b6c77a8123fdb062, State: Initialized, Role: FOLLOWER
I20260927 00:47:24.466578 7773 consensus_queue.cc:260] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4312a0972ce4ab3905b603a432eacb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } } peers { permanent_uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34123 } } peers { permanent_uuid: "8ed31e81e1864605b6c77a8123fdb062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41565 } }
I20260927 00:47:24.466876 7774 raft_consensus.cc:348] T eeaf1897e87d4b348e362db9a813e552 P f4312a0972ce4ab3905b603a432eacb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4312a0972ce4ab3905b603a432eacb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } } peers { permanent_uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34123 } } peers { permanent_uuid: "8ed31e81e1864605b6c77a8123fdb062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41565 } }
I20260927 00:47:24.467185 7774 raft_consensus.cc:374] T eeaf1897e87d4b348e362db9a813e552 P f4312a0972ce4ab3905b603a432eacb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:24.467235 7774 raft_consensus.cc:729] T eeaf1897e87d4b348e362db9a813e552 P f4312a0972ce4ab3905b603a432eacb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4312a0972ce4ab3905b603a432eacb8, State: Initialized, Role: FOLLOWER
I20260927 00:47:24.467757 7774 consensus_queue.cc:260] T eeaf1897e87d4b348e362db9a813e552 P f4312a0972ce4ab3905b603a432eacb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4312a0972ce4ab3905b603a432eacb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } } peers { permanent_uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34123 } } peers { permanent_uuid: "8ed31e81e1864605b6c77a8123fdb062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41565 } }
I20260927 00:47:24.467873 7773 ts_tablet_manager.cc:1428] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:47:24.468941 7774 ts_tablet_manager.cc:1428] T eeaf1897e87d4b348e362db9a813e552 P f4312a0972ce4ab3905b603a432eacb8: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260927 00:47:24.469065 7465 heartbeater.cc:503] Master 127.0.98.62:34945 was elected leader, sending a full tablet report...
I20260927 00:47:24.469558 7777 tablet_bootstrap.cc:492] T eeaf1897e87d4b348e362db9a813e552 P 1fedc53da95e4fc5bcdc6e2a3ee28a86: Bootstrap starting.
I20260927 00:47:24.470675 7740 heartbeater.cc:503] Master 127.0.98.62:34945 was elected leader, sending a full tablet report...
I20260927 00:47:24.473115 7777 tablet_bootstrap.cc:654] T eeaf1897e87d4b348e362db9a813e552 P 1fedc53da95e4fc5bcdc6e2a3ee28a86: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:24.474277 7777 log.cc:824] T eeaf1897e87d4b348e362db9a813e552 P 1fedc53da95e4fc5bcdc6e2a3ee28a86: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:24.478039 7777 tablet_bootstrap.cc:492] T eeaf1897e87d4b348e362db9a813e552 P 1fedc53da95e4fc5bcdc6e2a3ee28a86: No bootstrap required, opened a new log
I20260927 00:47:24.478278 7777 ts_tablet_manager.cc:1397] T eeaf1897e87d4b348e362db9a813e552 P 1fedc53da95e4fc5bcdc6e2a3ee28a86: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260927 00:47:24.482481 7777 raft_consensus.cc:348] T eeaf1897e87d4b348e362db9a813e552 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4312a0972ce4ab3905b603a432eacb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } } peers { permanent_uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34123 } } peers { permanent_uuid: "8ed31e81e1864605b6c77a8123fdb062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41565 } }
I20260927 00:47:24.482836 7777 raft_consensus.cc:374] T eeaf1897e87d4b348e362db9a813e552 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:24.482877 7777 raft_consensus.cc:729] T eeaf1897e87d4b348e362db9a813e552 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fedc53da95e4fc5bcdc6e2a3ee28a86, State: Initialized, Role: FOLLOWER
I20260927 00:47:24.483505 7777 consensus_queue.cc:260] T eeaf1897e87d4b348e362db9a813e552 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4312a0972ce4ab3905b603a432eacb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } } peers { permanent_uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34123 } } peers { permanent_uuid: "8ed31e81e1864605b6c77a8123fdb062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41565 } }
I20260927 00:47:24.484490 7602 heartbeater.cc:503] Master 127.0.98.62:34945 was elected leader, sending a full tablet report...
I20260927 00:47:24.484969 7777 ts_tablet_manager.cc:1428] T eeaf1897e87d4b348e362db9a813e552 P 1fedc53da95e4fc5bcdc6e2a3ee28a86: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260927 00:47:24.582072 7778 raft_consensus.cc:482] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:47:24.582382 7778 raft_consensus.cc:504] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4312a0972ce4ab3905b603a432eacb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } } peers { permanent_uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34123 } } peers { permanent_uuid: "8ed31e81e1864605b6c77a8123fdb062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41565 } }
I20260927 00:47:24.583873 7778 leader_election.cc:288] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4312a0972ce4ab3905b603a432eacb8 (127.0.98.3:42375), 1fedc53da95e4fc5bcdc6e2a3ee28a86 (127.0.98.2:34123)
I20260927 00:47:24.589705 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.584311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50819 (local address 127.0.98.3:42375)
0927 00:47:24.584617 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:24.584622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:24.584692 (+ 70us) server_negotiation.cc:407] Connection header received
0927 00:47:24.584890 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:24.584895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:24.584962 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:24.585058 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:24.585497 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.586357 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.588330 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.588971 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.589088 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.589191 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:24.589461 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 00:47:24.589544 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198}
I20260927 00:47:24.589705 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.584448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38321 (local address 127.0.98.2:34123)
0927 00:47:24.584582 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:24.584588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:24.584853 (+ 265us) server_negotiation.cc:407] Connection header received
0927 00:47:24.584923 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:24.584926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:24.584984 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:24.585065 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:24.585497 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.586337 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.588358 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.588971 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.589088 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.589167 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:24.589461 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 00:47:24.589544 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:47:24.589774 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.584181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34123 (local address 127.0.98.1:38321)
0927 00:47:24.584723 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:24.584736 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:24.584831 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:24.585093 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:24.585096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:24.585126 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:24.585368 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:24.585375 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.586469 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.586472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:24.588164 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:24.588177 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.589108 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.589111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:24.589296 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.589348 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 00:47:24.589431 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 00:47:24.589554 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":308,"spinlock_wait_cycles":6272,"thread_start_us":131,"threads_started":1}
I20260927 00:47:24.589963 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.584181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42375 (local address 127.0.98.1:50819)
0927 00:47:24.584575 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:24.584592 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:24.584747 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:24.585093 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:24.585096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:24.585134 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:24.585368 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:24.585375 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.586469 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.586472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:24.588171 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:24.588177 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.589108 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.589111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:24.589319 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.589353 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 00:47:24.589431 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 00:47:24.589555 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":348,"spinlock_wait_cycles":4736,"thread_start_us":166,"threads_started":1}
I20260927 00:47:24.590574 7555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeaf1897e87d4b348e362db9a813e552" candidate_uuid: "8ed31e81e1864605b6c77a8123fdb062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86" is_pre_election: true
I20260927 00:47:24.590983 7555 raft_consensus.cc:2454] T eeaf1897e87d4b348e362db9a813e552 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ed31e81e1864605b6c77a8123fdb062 in term 0.
I20260927 00:47:24.591063 7693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeaf1897e87d4b348e362db9a813e552" candidate_uuid: "8ed31e81e1864605b6c77a8123fdb062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4312a0972ce4ab3905b603a432eacb8" is_pre_election: true
I20260927 00:47:24.591444 7693 raft_consensus.cc:2454] T eeaf1897e87d4b348e362db9a813e552 P f4312a0972ce4ab3905b603a432eacb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ed31e81e1864605b6c77a8123fdb062 in term 0.
I20260927 00:47:24.591557 7352 leader_election.cc:302] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fedc53da95e4fc5bcdc6e2a3ee28a86, 8ed31e81e1864605b6c77a8123fdb062; no voters:
I20260927 00:47:24.591909 7778 raft_consensus.cc:2781] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:47:24.592002 7778 raft_consensus.cc:482] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:47:24.592046 7778 raft_consensus.cc:3037] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:24.593395 7778 raft_consensus.cc:504] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4312a0972ce4ab3905b603a432eacb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } } peers { permanent_uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34123 } } peers { permanent_uuid: "8ed31e81e1864605b6c77a8123fdb062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41565 } }
I20260927 00:47:24.593791 7778 leader_election.cc:288] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062 [CANDIDATE]: Term 1 election: Requested vote from peers f4312a0972ce4ab3905b603a432eacb8 (127.0.98.3:42375), 1fedc53da95e4fc5bcdc6e2a3ee28a86 (127.0.98.2:34123)
I20260927 00:47:24.594190 7555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeaf1897e87d4b348e362db9a813e552" candidate_uuid: "8ed31e81e1864605b6c77a8123fdb062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86"
I20260927 00:47:24.594192 7693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeaf1897e87d4b348e362db9a813e552" candidate_uuid: "8ed31e81e1864605b6c77a8123fdb062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4312a0972ce4ab3905b603a432eacb8"
I20260927 00:47:24.594350 7693 raft_consensus.cc:3037] T eeaf1897e87d4b348e362db9a813e552 P f4312a0972ce4ab3905b603a432eacb8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:24.594342 7555 raft_consensus.cc:3037] T eeaf1897e87d4b348e362db9a813e552 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:24.595597 7555 raft_consensus.cc:2454] T eeaf1897e87d4b348e362db9a813e552 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ed31e81e1864605b6c77a8123fdb062 in term 1.
I20260927 00:47:24.595904 7352 leader_election.cc:302] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fedc53da95e4fc5bcdc6e2a3ee28a86, 8ed31e81e1864605b6c77a8123fdb062; no voters:
I20260927 00:47:24.596104 7778 raft_consensus.cc:2781] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:24.596414 7778 raft_consensus.cc:686] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062 [term 1 LEADER]: Becoming Leader. State: Replica: 8ed31e81e1864605b6c77a8123fdb062, State: Running, Role: LEADER
I20260927 00:47:24.596585 7693 raft_consensus.cc:2454] T eeaf1897e87d4b348e362db9a813e552 P f4312a0972ce4ab3905b603a432eacb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ed31e81e1864605b6c77a8123fdb062 in term 1.
I20260927 00:47:24.596921 7778 consensus_queue.cc:237] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4312a0972ce4ab3905b603a432eacb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } } peers { permanent_uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34123 } } peers { permanent_uuid: "8ed31e81e1864605b6c77a8123fdb062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41565 } }
I20260927 00:47:24.600944 6500 catalog_manager.cc:5495] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ed31e81e1864605b6c77a8123fdb062 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8ed31e81e1864605b6c77a8123fdb062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4312a0972ce4ab3905b603a432eacb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ed31e81e1864605b6c77a8123fdb062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41565 } health_report { overall_health: HEALTHY } } }
W20260927 00:47:24.602240 7604 tablet.cc:2369] T eeaf1897e87d4b348e362db9a813e552 P 1fedc53da95e4fc5bcdc6e2a3ee28a86: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:47:24.604791 7467 tablet.cc:2369] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:47:24.622043 7742 tablet.cc:2369] T eeaf1897e87d4b348e362db9a813e552 P f4312a0972ce4ab3905b603a432eacb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:47:24.629298 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.624903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40760 (local address 127.0.98.1:41565)
0927 00:47:24.625050 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:24.625054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:24.625069 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:47:24.625175 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:24.625177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:24.625245 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:24.625359 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:47:24.625656 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.626306 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.627295 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.627560 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.627733 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.628074 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:47:24.628842 (+ 768us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:47:24.629008 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:24.629056 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 00:47:24.629137 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:47:24.638033 7693 raft_consensus.cc:1261] T eeaf1897e87d4b348e362db9a813e552 P f4312a0972ce4ab3905b603a432eacb8 [term 1 FOLLOWER]: Refusing update from remote peer 8ed31e81e1864605b6c77a8123fdb062: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:47:24.638034 7555 raft_consensus.cc:1261] T eeaf1897e87d4b348e362db9a813e552 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [term 1 FOLLOWER]: Refusing update from remote peer 8ed31e81e1864605b6c77a8123fdb062: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:47:24.639223 7784 consensus_queue.cc:1035] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:47:24.639559 7778 consensus_queue.cc:1035] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4312a0972ce4ab3905b603a432eacb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:47:24.658455 7789 mvcc.cc:205] Tried to move back new op lower bound from 7333765302826254336 to 7333765302672662528. Current Snapshot: MvccSnapshot[applied={T|T < 7333765302826254336}]
I20260927 00:47:24.671272 6498 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:47:24.671589 6498 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:47:24.679004 6498 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5259924bdea44542b5131a31ce919b1d: Sending DeleteTablet for 3 replicas of tablet eeaf1897e87d4b348e362db9a813e552
I20260927 00:47:24.680101 7673 tablet_service.cc:1514] Processing DeleteTablet for tablet eeaf1897e87d4b348e362db9a813e552 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56478
I20260927 00:47:24.680505 7398 tablet_service.cc:1514] Processing DeleteTablet for tablet eeaf1897e87d4b348e362db9a813e552 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40746
I20260927 00:47:24.681013 7797 tablet_replica.cc:330] T eeaf1897e87d4b348e362db9a813e552 P f4312a0972ce4ab3905b603a432eacb8: stopping tablet replica
I20260927 00:47:24.680958 7535 tablet_service.cc:1514] Processing DeleteTablet for tablet eeaf1897e87d4b348e362db9a813e552 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40108
I20260927 00:47:24.681509 7799 tablet_replica.cc:330] T eeaf1897e87d4b348e362db9a813e552 P 1fedc53da95e4fc5bcdc6e2a3ee28a86: stopping tablet replica
I20260927 00:47:24.681716 7797 raft_consensus.cc:2229] T eeaf1897e87d4b348e362db9a813e552 P f4312a0972ce4ab3905b603a432eacb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:47:24.681919 7799 raft_consensus.cc:2229] T eeaf1897e87d4b348e362db9a813e552 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:47:24.682307 7797 raft_consensus.cc:2258] T eeaf1897e87d4b348e362db9a813e552 P f4312a0972ce4ab3905b603a432eacb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:24.682312 7799 raft_consensus.cc:2258] T eeaf1897e87d4b348e362db9a813e552 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:24.684823 7797 ts_tablet_manager.cc:1905] T eeaf1897e87d4b348e362db9a813e552 P f4312a0972ce4ab3905b603a432eacb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:24.684823 7799 ts_tablet_manager.cc:1905] T eeaf1897e87d4b348e362db9a813e552 P 1fedc53da95e4fc5bcdc6e2a3ee28a86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:24.687821 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 6468
I20260927 00:47:24.689070 7799 ts_tablet_manager.cc:1918] T eeaf1897e87d4b348e362db9a813e552 P 1fedc53da95e4fc5bcdc6e2a3ee28a86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:47:24.689203 7799 log.cc:1196] T eeaf1897e87d4b348e362db9a813e552 P 1fedc53da95e4fc5bcdc6e2a3ee28a86: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/wal/wals/eeaf1897e87d4b348e362db9a813e552
I20260927 00:47:24.689600 7799 ts_tablet_manager.cc:1939] T eeaf1897e87d4b348e362db9a813e552 P 1fedc53da95e4fc5bcdc6e2a3ee28a86: Deleting consensus metadata
I20260927 00:47:24.691582 7797 ts_tablet_manager.cc:1918] T eeaf1897e87d4b348e362db9a813e552 P f4312a0972ce4ab3905b603a432eacb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:47:24.691994 7797 log.cc:1196] T eeaf1897e87d4b348e362db9a813e552 P f4312a0972ce4ab3905b603a432eacb8: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/wal/wals/eeaf1897e87d4b348e362db9a813e552
I20260927 00:47:24.692453 7797 ts_tablet_manager.cc:1939] T eeaf1897e87d4b348e362db9a813e552 P f4312a0972ce4ab3905b603a432eacb8: Deleting consensus metadata
I20260927 00:47:24.696589 7798 tablet_replica.cc:330] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062: stopping tablet replica
I20260927 00:47:24.697242 7798 raft_consensus.cc:2229] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062 [term 1 LEADER]: Raft consensus shutting down.
W20260927 00:47:24.697492 7488 connection.cc:535] server connection from 127.0.0.1:40108 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 00:47:24.698060 7626 connection.cc:535] server connection from 127.0.0.1:56478 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 00:47:24.700157 7798 raft_consensus.cc:2258] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:24.701901 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.685394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34258 (local address 127.0.98.61:34013)
0927 00:47:24.685574 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:24.685580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:24.689138 (+ 3558us) server_negotiation.cc:407] Connection header received
0927 00:47:24.689570 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:24.689574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:24.689661 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:24.689721 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:24.691959 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.692991 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.694488 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.694904 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.695025 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.697466 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:24.697471 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:24.697474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:24.697520 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:24.699769 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:24.700405 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:24.700409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:24.700411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:24.700508 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:24.700851 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:24.700855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:24.700857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:24.701127 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:24.701270 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:24.701548 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 00:47:24.701709 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 00:47:24.704069 7798 ts_tablet_manager.cc:1905] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:24.707803 7798 ts_tablet_manager.cc:1918] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:47:24.707978 7798 log.cc:1196] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-0/wal/wals/eeaf1897e87d4b348e362db9a813e552
I20260927 00:47:24.708462 7798 ts_tablet_manager.cc:1939] T eeaf1897e87d4b348e362db9a813e552 P 8ed31e81e1864605b6c77a8123fdb062: Deleting consensus metadata
W20260927 00:47:24.710214 7351 connection.cc:441] server connection from 127.0.0.1:40746 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40746 (request call id 1) could send its response
I20260927 00:47:24.711776 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.685708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37656 (local address 127.0.98.60:32877)
0927 00:47:24.685850 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:24.685855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:24.701943 (+ 16088us) server_negotiation.cc:407] Connection header received
0927 00:47:24.702082 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:24.702088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:24.702168 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:24.702247 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:24.703165 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.704056 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.705157 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.705548 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.705679 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.707464 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:24.707474 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:24.707478 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:24.707515 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:24.709519 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:24.710070 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:24.710075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:24.710076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:24.710181 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:24.710525 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:24.710531 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:24.710536 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:24.710813 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:24.710947 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:24.711328 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 00:47:24.711483 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:47:24.721334 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.709928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34266 (local address 127.0.98.61:34013)
0927 00:47:24.710074 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:24.710079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:24.712561 (+ 2482us) server_negotiation.cc:407] Connection header received
0927 00:47:24.712748 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:24.712751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:24.712811 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:24.712867 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:24.713772 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.714339 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.715386 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.715750 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.715853 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.717638 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:24.717643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:24.717646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:24.717685 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:24.719309 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:24.719800 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:24.719805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:24.719807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:24.719912 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:24.720279 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:24.720283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:24.720285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:24.720539 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:24.720658 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:24.721002 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 00:47:24.721143 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:47:24.721334 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:24.708825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37660 (local address 127.0.98.60:32877)
0927 00:47:24.709009 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:24.709013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:24.711169 (+ 2156us) server_negotiation.cc:407] Connection header received
0927 00:47:24.711355 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:24.711358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:24.711412 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:24.711462 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:24.713038 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.713885 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.715182 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:24.715496 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:24.715615 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:24.717840 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:24.717844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:24.717848 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:24.717878 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:24.719543 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:24.719978 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:24.719981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:24.719983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:24.720072 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:24.720391 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:24.720394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:24.720396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:24.720603 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:24.720748 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:24.720991 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 00:47:24.721124 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 00:47:25.658792 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:25.657839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.98.1:52455)
0927 00:47:25.658478 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:25.658607 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":99,"threads_started":1}
I20260927 00:47:25.658895 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:25.657333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.98.2:38991)
0927 00:47:25.658634 (+ 1301us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:25.658746 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1194,"thread_start_us":106,"threads_started":1}
W20260927 00:47:25.659204 7465 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34945 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
W20260927 00:47:25.659364 7602 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34945 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
I20260927 00:47:25.659421 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:25.658503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.98.3:50023)
0927 00:47:25.659197 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:25.659302 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":79,"threads_started":1}
W20260927 00:47:25.659662 7740 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34945 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
I20260927 00:47:26.178948 7811 raft_consensus.cc:482] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5259924bdea44542b5131a31ce919b1d)
I20260927 00:47:26.179055 7812 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5259924bdea44542b5131a31ce919b1d)
I20260927 00:47:26.179291 7812 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: false } }
I20260927 00:47:26.179262 7811 raft_consensus.cc:504] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: false } }
I20260927 00:47:26.180655 7812 leader_election.cc:288] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5259924bdea44542b5131a31ce919b1d (127.0.98.62:34945), ceb92bf11e1b41a19b0118bb8a88002d (127.0.98.61:34013)
I20260927 00:47:26.180661 7811 leader_election.cc:288] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5259924bdea44542b5131a31ce919b1d (127.0.98.62:34945), e3bfc9d2ca344168be147ebe68ef5d31 (127.0.98.60:32877)
I20260927 00:47:26.181545 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:26.181018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.0.1:50256)
0927 00:47:26.181309 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:26.181404 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":104,"threads_started":1}
I20260927 00:47:26.181599 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:26.180878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.0.1:50254)
0927 00:47:26.181395 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:26.181487 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":91,"threads_started":1}
W20260927 00:47:26.181767 7086 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
W20260927 00:47:26.181751 7270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
I20260927 00:47:26.182875 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:26.182588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.0.1:50266)
0927 00:47:26.182727 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:26.182787 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 00:47:26.183218 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:26.182961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.0.1:50272)
0927 00:47:26.183085 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:26.183145 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260927 00:47:26.183377 7086 leader_election.cc:334] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5259924bdea44542b5131a31ce919b1d (127.0.98.62:34945): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
W20260927 00:47:26.183600 7270 leader_election.cc:334] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5259924bdea44542b5131a31ce919b1d (127.0.98.62:34945): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
I20260927 00:47:26.185928 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:26.181099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37674 (local address 127.0.98.60:32877)
0927 00:47:26.181328 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:26.181333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:26.181534 (+ 201us) server_negotiation.cc:407] Connection header received
0927 00:47:26.181790 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:26.181794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:26.181855 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:26.181924 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:26.182428 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:26.183268 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:26.184621 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:26.185314 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:26.185407 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:26.185468 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:26.185751 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 00:47:26.185831 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":160,"thread_start_us":77,"threads_started":1}
I20260927 00:47:26.185945 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:26.181103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32877 (local address 127.0.0.1:37674)
0927 00:47:26.181438 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:26.181453 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:26.181642 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:26.182114 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:26.182117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:26.182141 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:26.182269 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:26.182275 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:26.183405 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:26.183408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:26.184501 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:26.184511 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:26.185419 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:26.185422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:26.185584 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:26.185613 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 00:47:26.185700 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 00:47:26.185783 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":43,"thread_start_us":70,"threads_started":1}
I20260927 00:47:26.186628 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:26.181613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34278 (local address 127.0.98.61:34013)
0927 00:47:26.182117 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:26.182122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:26.182331 (+ 209us) server_negotiation.cc:407] Connection header received
0927 00:47:26.182485 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:26.182491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:26.182545 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:26.182598 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:26.183014 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:26.183886 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:26.185310 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:26.185997 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:26.186105 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:26.186167 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:26.186472 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 00:47:26.186540 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":431,"thread_start_us":70,"threads_started":1}
I20260927 00:47:26.186533 7295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" is_pre_election: true
I20260927 00:47:26.186678 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:26.181358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34013 (local address 127.0.0.1:34278)
0927 00:47:26.182222 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:26.182236 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:26.182356 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:26.182768 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:26.182771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:26.182793 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:26.182887 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:26.182898 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:26.183993 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:26.183998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:26.185187 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:26.185197 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:26.186233 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:26.186236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:26.186338 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:26.186362 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:47:26.186440 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 00:47:26.186514 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":22,"thread_start_us":62,"threads_started":1}
I20260927 00:47:26.186839 7295 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb92bf11e1b41a19b0118bb8a88002d in term 1.
I20260927 00:47:26.187170 7111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" is_pre_election: true
I20260927 00:47:26.187275 7085 leader_election.cc:302] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ceb92bf11e1b41a19b0118bb8a88002d, e3bfc9d2ca344168be147ebe68ef5d31; no voters: 5259924bdea44542b5131a31ce919b1d
I20260927 00:47:26.187428 7111 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3bfc9d2ca344168be147ebe68ef5d31 in term 1.
I20260927 00:47:26.187605 7811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 00:47:26.187669 7811 raft_consensus.cc:482] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5259924bdea44542b5131a31ce919b1d)
I20260927 00:47:26.187712 7811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:47:26.187784 7270 leader_election.cc:302] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ceb92bf11e1b41a19b0118bb8a88002d, e3bfc9d2ca344168be147ebe68ef5d31; no voters: 5259924bdea44542b5131a31ce919b1d
I20260927 00:47:26.188009 7812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 00:47:26.188057 7812 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5259924bdea44542b5131a31ce919b1d)
I20260927 00:47:26.188086 7812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:47:26.189150 7811 raft_consensus.cc:504] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: false } }
I20260927 00:47:26.189338 7812 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: false } }
I20260927 00:47:26.189652 7811 leader_election.cc:288] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [CANDIDATE]: Term 2 election: Requested vote from peers 5259924bdea44542b5131a31ce919b1d (127.0.98.62:34945), e3bfc9d2ca344168be147ebe68ef5d31 (127.0.98.60:32877)
I20260927 00:47:26.189805 7812 leader_election.cc:288] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [CANDIDATE]: Term 2 election: Requested vote from peers 5259924bdea44542b5131a31ce919b1d (127.0.98.62:34945), ceb92bf11e1b41a19b0118bb8a88002d (127.0.98.61:34013)
I20260927 00:47:26.189978 7295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e3bfc9d2ca344168be147ebe68ef5d31"
I20260927 00:47:26.190116 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:26.189807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.0.1:50288)
0927 00:47:26.189982 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:26.190044 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260927 00:47:26.190155 7295 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ceb92bf11e1b41a19b0118bb8a88002d in current term 2: Already voted for candidate e3bfc9d2ca344168be147ebe68ef5d31 in this term.
I20260927 00:47:26.190246 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:26.189986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.0.1:50292)
0927 00:47:26.190098 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:26.190152 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 00:47:26.190353 7111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ceb92bf11e1b41a19b0118bb8a88002d"
I20260927 00:47:26.190513 7111 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e3bfc9d2ca344168be147ebe68ef5d31 in current term 2: Already voted for candidate ceb92bf11e1b41a19b0118bb8a88002d in this term.
I20260927 00:47:26.190779 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:26.190569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.0.1:50298)
0927 00:47:26.190674 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:26.190723 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 00:47:26.191105 7086 leader_election.cc:334] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5259924bdea44542b5131a31ce919b1d (127.0.98.62:34945): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
I20260927 00:47:26.191176 7086 leader_election.cc:302] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ceb92bf11e1b41a19b0118bb8a88002d; no voters: 5259924bdea44542b5131a31ce919b1d, e3bfc9d2ca344168be147ebe68ef5d31
I20260927 00:47:26.191329 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:26.191096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.0.1:50308)
0927 00:47:26.191205 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:26.191259 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 00:47:26.191335 7811 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260927 00:47:26.191483 7270 leader_election.cc:334] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5259924bdea44542b5131a31ce919b1d (127.0.98.62:34945): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
I20260927 00:47:26.191551 7270 leader_election.cc:302] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3bfc9d2ca344168be147ebe68ef5d31; no voters: 5259924bdea44542b5131a31ce919b1d, ceb92bf11e1b41a19b0118bb8a88002d
I20260927 00:47:26.191707 7812 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 00:47:26.660758 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:26.660132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.98.1:48565)
0927 00:47:26.660523 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:26.660609 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":171,"threads_started":1}
I20260927 00:47:26.660830 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:26.660263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.98.3:40643)
0927 00:47:26.660624 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:26.660697 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":95,"threads_started":1}
I20260927 00:47:26.661433 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:26.660461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.98.2:53293)
0927 00:47:26.661234 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:26.661308 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":80,"threads_started":1}
I20260927 00:47:27.662621 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.661712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.98.3:56609)
0927 00:47:27.662379 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:27.662470 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":102,"threads_started":1}
I20260927 00:47:27.662698 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.661990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.98.1:42121)
0927 00:47:27.662496 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:27.662557 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":86,"threads_started":1}
I20260927 00:47:27.663532 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.662133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.98.2:32973)
0927 00:47:27.663352 (+ 1219us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:27.663421 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1132,"thread_start_us":68,"threads_started":1}
I20260927 00:47:27.715740 7827 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:47:27.715960 7827 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: false } }
I20260927 00:47:27.716563 7827 leader_election.cc:288] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5259924bdea44542b5131a31ce919b1d (127.0.98.62:34945), ceb92bf11e1b41a19b0118bb8a88002d (127.0.98.61:34013)
I20260927 00:47:27.717382 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.716863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.0.1:50502)
0927 00:47:27.717174 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:27.717261 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":141,"threads_started":1}
I20260927 00:47:27.717674 7111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" is_pre_election: true
I20260927 00:47:27.717890 7111 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3bfc9d2ca344168be147ebe68ef5d31 in term 2.
I20260927 00:47:27.718305 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.718058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.0.1:50518)
0927 00:47:27.718172 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:27.718229 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 00:47:27.718267 7270 leader_election.cc:302] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ceb92bf11e1b41a19b0118bb8a88002d, e3bfc9d2ca344168be147ebe68ef5d31; no voters:
I20260927 00:47:27.718463 7827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260927 00:47:27.718464 7270 leader_election.cc:334] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5259924bdea44542b5131a31ce919b1d (127.0.98.62:34945): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
I20260927 00:47:27.718541 7827 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:47:27.718582 7827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [term 2 FOLLOWER]: Advancing to term 3
I20260927 00:47:27.719832 7827 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: false } }
I20260927 00:47:27.720232 7827 leader_election.cc:288] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [CANDIDATE]: Term 3 election: Requested vote from peers 5259924bdea44542b5131a31ce919b1d (127.0.98.62:34945), ceb92bf11e1b41a19b0118bb8a88002d (127.0.98.61:34013)
I20260927 00:47:27.720638 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.720380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.0.1:50524)
0927 00:47:27.720493 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:27.720551 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 00:47:27.720904 7111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ceb92bf11e1b41a19b0118bb8a88002d"
I20260927 00:47:27.721077 7111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [term 2 FOLLOWER]: Advancing to term 3
I20260927 00:47:27.721495 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.721246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.0.1:50530)
0927 00:47:27.721369 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:27.721424 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 00:47:27.721616 7270 leader_election.cc:334] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 5259924bdea44542b5131a31ce919b1d (127.0.98.62:34945): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
I20260927 00:47:27.722445 7111 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3bfc9d2ca344168be147ebe68ef5d31 in term 3.
I20260927 00:47:27.722798 7270 leader_election.cc:302] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ceb92bf11e1b41a19b0118bb8a88002d, e3bfc9d2ca344168be147ebe68ef5d31; no voters: 5259924bdea44542b5131a31ce919b1d
I20260927 00:47:27.722992 7827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 00:47:27.723212 7827 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [term 3 LEADER]: Becoming Leader. State: Replica: e3bfc9d2ca344168be147ebe68ef5d31, State: Running, Role: LEADER
I20260927 00:47:27.723542 7827 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: false } }
I20260927 00:47:27.725073 7830 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3bfc9d2ca344168be147ebe68ef5d31. Latest consensus state: current_term: 3 leader_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: false } } }
I20260927 00:47:27.725214 7830 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:27.725567 7832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:47:27.728436 7832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:47:27.728929 7832 catalog_manager.cc:1259] Loaded cluster ID: 6864942181e149ff81377ee6995feab8
I20260927 00:47:27.728986 7832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:47:27.729664 7832 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:47:27.730108 7832 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31: Loaded TSK: 0
I20260927 00:47:27.731140 7832 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:47:27.744772 7285 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:47:27.748920 7285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:47:27.752931 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.752588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.0.1:50544)
0927 00:47:27.752723 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:27.752818 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 00:47:27.753367 7111 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [term 3 FOLLOWER]: Refusing update from remote peer e3bfc9d2ca344168be147ebe68ef5d31: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 00:47:27.753822 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.753543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.0.1:50548)
0927 00:47:27.753638 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:27.753732 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 00:47:27.754006 7270 consensus_peers.cc:487] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 -> Peer 5259924bdea44542b5131a31ce919b1d (127.0.98.62:34945): Couldn't send request to peer 5259924bdea44542b5131a31ce919b1d. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 00:47:27.754020 7827 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 00:47:27.755918 7834 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3bfc9d2ca344168be147ebe68ef5d31. Latest consensus state: current_term: 3 leader_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: false } } }
I20260927 00:47:27.756160 7834 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:47:27.757539 7833 mvcc.cc:205] Tried to move back new op lower bound from 7333765315589533696 to 7333765315478945792. Current Snapshot: MvccSnapshot[applied={T|T < 7333765315589533696}]
I20260927 00:47:27.757705 7827 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: false } } }
I20260927 00:47:27.757859 7827 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31 [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:27.758198 7834 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5259924bdea44542b5131a31ce919b1d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } peers { permanent_uuid: "ceb92bf11e1b41a19b0118bb8a88002d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34013 } attrs { promote: false } } peers { permanent_uuid: "e3bfc9d2ca344168be147ebe68ef5d31" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32877 } attrs { promote: false } } }
I20260927 00:47:27.758355 7834 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb92bf11e1b41a19b0118bb8a88002d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:47:27.771576 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.766013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34123 (local address 127.0.0.1:40120)
0927 00:47:27.766551 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:27.766569 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:27.766694 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:27.767382 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:27.767384 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:27.767416 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:27.767646 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:27.767655 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:27.768783 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:27.768785 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:27.770178 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:27.770187 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:27.771131 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:27.771134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:27.771224 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:27.771250 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:47:27.771299 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 00:47:27.771399 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":180,"spinlock_wait_cycles":2816}
I20260927 00:47:27.771586 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.766557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56484 (local address 127.0.98.3:42375)
0927 00:47:27.767196 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:27.767201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:27.767215 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:47:27.767274 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:27.767276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:27.767341 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:27.767431 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:27.767772 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:27.768560 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:27.770420 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:27.770951 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:27.771076 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:27.771169 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:27.771275 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 00:47:27.771354 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":95,"threads_started":1}
I20260927 00:47:27.771579 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.766420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42375 (local address 127.0.0.1:56484)
0927 00:47:27.766842 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:27.766857 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:27.766953 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:27.767449 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:27.767452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:27.767464 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:27.767619 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:27.767627 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:27.768690 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:27.768693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:27.770321 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:27.770330 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:27.771069 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:27.771073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:27.771158 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:27.771191 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 00:47:27.771263 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 00:47:27.771354 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":126,"thread_start_us":98,"threads_started":1}
I20260927 00:47:27.771832 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.766160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40120 (local address 127.0.98.2:34123)
0927 00:47:27.767112 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:27.767117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:27.767132 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:47:27.767205 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:27.767211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:27.767270 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:27.767367 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:27.767781 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:27.768631 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:27.770282 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:27.770991 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:27.771390 (+ 399us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:27.771520 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:27.771558 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 00:47:27.771670 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":857,"thread_start_us":102,"threads_started":1}
I20260927 00:47:27.772066 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.766269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41565 (local address 127.0.0.1:40770)
0927 00:47:27.766671 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:27.766682 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:27.766766 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:27.767980 (+ 1214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:27.767983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:27.768001 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:27.768102 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:27.768108 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:27.769068 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:27.769073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:27.770203 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:27.770213 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:27.771256 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:27.771259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:27.771776 (+ 517us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:27.771799 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:47:27.771867 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 00:47:27.771949 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":27,"thread_start_us":103,"threads_started":1}
I20260927 00:47:27.772539 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.766408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40770 (local address 127.0.98.1:41565)
0927 00:47:27.767691 (+ 1283us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:27.767695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:27.767740 (+ 45us) server_negotiation.cc:407] Connection header received
0927 00:47:27.767802 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:27.767805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:27.767862 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:27.767959 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:27.768268 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:27.768963 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:27.770340 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:27.770995 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:27.771108 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:27.772219 (+ 1111us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:27.772290 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 00:47:27.772394 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1191,"thread_start_us":88,"threads_started":1}
I20260927 00:47:27.772470 7535 tablet_service.cc:1467] Processing CreateTablet for tablet 722061464c204e8bac63eb066d1acc92 (DEFAULT_TABLE table=test-table [id=5e75f99151e446cabafd6abd56b0619d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:27.772904 7535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 722061464c204e8bac63eb066d1acc92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:27.772846 7673 tablet_service.cc:1467] Processing CreateTablet for tablet 722061464c204e8bac63eb066d1acc92 (DEFAULT_TABLE table=test-table [id=5e75f99151e446cabafd6abd56b0619d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:27.773152 7673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 722061464c204e8bac63eb066d1acc92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:27.773234 7398 tablet_service.cc:1467] Processing CreateTablet for tablet 722061464c204e8bac63eb066d1acc92 (DEFAULT_TABLE table=test-table [id=5e75f99151e446cabafd6abd56b0619d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:27.773522 7398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 722061464c204e8bac63eb066d1acc92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:27.775671 7845 tablet_bootstrap.cc:492] T 722061464c204e8bac63eb066d1acc92 P f4312a0972ce4ab3905b603a432eacb8: Bootstrap starting.
I20260927 00:47:27.777026 7846 tablet_bootstrap.cc:492] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86: Bootstrap starting.
I20260927 00:47:27.777534 7845 tablet_bootstrap.cc:654] T 722061464c204e8bac63eb066d1acc92 P f4312a0972ce4ab3905b603a432eacb8: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:27.778122 7846 tablet_bootstrap.cc:654] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:27.778373 7847 tablet_bootstrap.cc:492] T 722061464c204e8bac63eb066d1acc92 P 8ed31e81e1864605b6c77a8123fdb062: Bootstrap starting.
I20260927 00:47:27.779014 7845 tablet_bootstrap.cc:492] T 722061464c204e8bac63eb066d1acc92 P f4312a0972ce4ab3905b603a432eacb8: No bootstrap required, opened a new log
I20260927 00:47:27.779124 7845 ts_tablet_manager.cc:1397] T 722061464c204e8bac63eb066d1acc92 P f4312a0972ce4ab3905b603a432eacb8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 00:47:27.779486 7846 tablet_bootstrap.cc:492] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86: No bootstrap required, opened a new log
I20260927 00:47:27.779587 7846 ts_tablet_manager.cc:1397] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:47:27.779692 7845 raft_consensus.cc:348] T 722061464c204e8bac63eb066d1acc92 P f4312a0972ce4ab3905b603a432eacb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34123 } } peers { permanent_uuid: "8ed31e81e1864605b6c77a8123fdb062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41565 } } peers { permanent_uuid: "f4312a0972ce4ab3905b603a432eacb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } }
I20260927 00:47:27.779840 7845 raft_consensus.cc:374] T 722061464c204e8bac63eb066d1acc92 P f4312a0972ce4ab3905b603a432eacb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:27.779882 7845 raft_consensus.cc:729] T 722061464c204e8bac63eb066d1acc92 P f4312a0972ce4ab3905b603a432eacb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4312a0972ce4ab3905b603a432eacb8, State: Initialized, Role: FOLLOWER
I20260927 00:47:27.780045 7845 consensus_queue.cc:260] T 722061464c204e8bac63eb066d1acc92 P f4312a0972ce4ab3905b603a432eacb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34123 } } peers { permanent_uuid: "8ed31e81e1864605b6c77a8123fdb062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41565 } } peers { permanent_uuid: "f4312a0972ce4ab3905b603a432eacb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } }
I20260927 00:47:27.780525 7845 ts_tablet_manager.cc:1428] T 722061464c204e8bac63eb066d1acc92 P f4312a0972ce4ab3905b603a432eacb8: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260927 00:47:27.781237 7847 tablet_bootstrap.cc:654] T 722061464c204e8bac63eb066d1acc92 P 8ed31e81e1864605b6c77a8123fdb062: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:27.781406 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.781113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.98.3:45987)
0927 00:47:27.781254 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:27.781318 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 00:47:27.782521 7847 tablet_bootstrap.cc:492] T 722061464c204e8bac63eb066d1acc92 P 8ed31e81e1864605b6c77a8123fdb062: No bootstrap required, opened a new log
I20260927 00:47:27.782625 7847 ts_tablet_manager.cc:1397] T 722061464c204e8bac63eb066d1acc92 P 8ed31e81e1864605b6c77a8123fdb062: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 00:47:27.783056 7847 raft_consensus.cc:348] T 722061464c204e8bac63eb066d1acc92 P 8ed31e81e1864605b6c77a8123fdb062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34123 } } peers { permanent_uuid: "8ed31e81e1864605b6c77a8123fdb062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41565 } } peers { permanent_uuid: "f4312a0972ce4ab3905b603a432eacb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } }
I20260927 00:47:27.783169 7847 raft_consensus.cc:374] T 722061464c204e8bac63eb066d1acc92 P 8ed31e81e1864605b6c77a8123fdb062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:27.783192 7847 raft_consensus.cc:729] T 722061464c204e8bac63eb066d1acc92 P 8ed31e81e1864605b6c77a8123fdb062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ed31e81e1864605b6c77a8123fdb062, State: Initialized, Role: FOLLOWER
I20260927 00:47:27.783361 7847 consensus_queue.cc:260] T 722061464c204e8bac63eb066d1acc92 P 8ed31e81e1864605b6c77a8123fdb062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34123 } } peers { permanent_uuid: "8ed31e81e1864605b6c77a8123fdb062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41565 } } peers { permanent_uuid: "f4312a0972ce4ab3905b603a432eacb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } }
I20260927 00:47:27.783816 7847 ts_tablet_manager.cc:1428] T 722061464c204e8bac63eb066d1acc92 P 8ed31e81e1864605b6c77a8123fdb062: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 00:47:27.783885 7846 raft_consensus.cc:348] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34123 } } peers { permanent_uuid: "8ed31e81e1864605b6c77a8123fdb062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41565 } } peers { permanent_uuid: "f4312a0972ce4ab3905b603a432eacb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } }
I20260927 00:47:27.784047 7846 raft_consensus.cc:374] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:27.784045 7739 heartbeater.cc:503] Master 127.0.98.60:32877 was elected leader, sending a full tablet report...
I20260927 00:47:27.784111 7846 raft_consensus.cc:729] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fedc53da95e4fc5bcdc6e2a3ee28a86, State: Initialized, Role: FOLLOWER
I20260927 00:47:27.784339 7846 consensus_queue.cc:260] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34123 } } peers { permanent_uuid: "8ed31e81e1864605b6c77a8123fdb062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41565 } } peers { permanent_uuid: "f4312a0972ce4ab3905b603a432eacb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } }
I20260927 00:47:27.784960 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.784693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.98.1:46855)
0927 00:47:27.784809 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:27.784872 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 00:47:27.785847 7846 ts_tablet_manager.cc:1428] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260927 00:47:27.785892 7464 heartbeater.cc:503] Master 127.0.98.60:32877 was elected leader, sending a full tablet report...
I20260927 00:47:27.786607 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.786358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.98.2:54317)
0927 00:47:27.786467 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:27.786528 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 00:47:27.787372 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.787092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.98.2:59857)
0927 00:47:27.787237 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:27.787301 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 00:47:27.787516 7601 heartbeater.cc:503] Master 127.0.98.60:32877 was elected leader, sending a full tablet report...
I20260927 00:47:27.788401 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.788118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.98.1:37175)
0927 00:47:27.788275 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:27.788331 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 00:47:27.802691 7853 raft_consensus.cc:482] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:47:27.802860 7853 raft_consensus.cc:504] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34123 } } peers { permanent_uuid: "8ed31e81e1864605b6c77a8123fdb062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41565 } } peers { permanent_uuid: "f4312a0972ce4ab3905b603a432eacb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } }
I20260927 00:47:27.804397 7853 leader_election.cc:288] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ed31e81e1864605b6c77a8123fdb062 (127.0.98.1:41565), f4312a0972ce4ab3905b603a432eacb8 (127.0.98.3:42375)
I20260927 00:47:27.808830 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.804764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60043 (local address 127.0.98.1:41565)
0927 00:47:27.804923 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:27.804929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:27.804947 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:47:27.805093 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:27.805097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:27.805147 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:27.805237 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:27.805528 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:27.806085 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:27.807645 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:27.808097 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:27.808283 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:27.808370 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:27.808565 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 00:47:27.808663 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:47:27.808830 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.804635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41565 (local address 127.0.98.2:60043)
0927 00:47:27.804807 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:27.804825 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:27.804957 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:27.805265 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:27.805268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:27.805290 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:27.805420 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:27.805426 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:27.806422 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:27.806429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:27.807514 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:27.807523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:27.808297 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:27.808301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:27.808417 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:27.808443 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:47:27.808548 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 00:47:27.808663 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":1}
I20260927 00:47:27.809010 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.804807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34735 (local address 127.0.98.3:42375)
0927 00:47:27.804949 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:27.804955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:27.805373 (+ 418us) server_negotiation.cc:407] Connection header received
0927 00:47:27.805522 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:27.805528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:27.805588 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:27.805686 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:27.805982 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:27.806688 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:27.807857 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:27.808384 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:27.808505 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:27.808581 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:27.808757 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 00:47:27.808855 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:47:27.809569 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.804639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42375 (local address 127.0.98.2:34735)
0927 00:47:27.805261 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:27.805273 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:27.805399 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:27.805694 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:27.805697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:27.805713 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:27.805839 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:27.805845 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:27.806803 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:27.806809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:27.807718 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:27.807727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:27.808505 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:27.808509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:27.808616 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:27.808638 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:47:27.808737 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 00:47:27.808814 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":190,"thread_start_us":191,"threads_started":1}
I20260927 00:47:27.809767 7418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "722061464c204e8bac63eb066d1acc92" candidate_uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ed31e81e1864605b6c77a8123fdb062" is_pre_election: true
I20260927 00:47:27.810086 7693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "722061464c204e8bac63eb066d1acc92" candidate_uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4312a0972ce4ab3905b603a432eacb8" is_pre_election: true
I20260927 00:47:27.810129 7418 raft_consensus.cc:2454] T 722061464c204e8bac63eb066d1acc92 P 8ed31e81e1864605b6c77a8123fdb062 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fedc53da95e4fc5bcdc6e2a3ee28a86 in term 0.
I20260927 00:47:27.810262 7693 raft_consensus.cc:2454] T 722061464c204e8bac63eb066d1acc92 P f4312a0972ce4ab3905b603a432eacb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fedc53da95e4fc5bcdc6e2a3ee28a86 in term 0.
I20260927 00:47:27.810717 7489 leader_election.cc:302] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1fedc53da95e4fc5bcdc6e2a3ee28a86, 8ed31e81e1864605b6c77a8123fdb062, f4312a0972ce4ab3905b603a432eacb8; no voters:
I20260927 00:47:27.811043 7853 raft_consensus.cc:2781] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:47:27.811102 7853 raft_consensus.cc:482] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:47:27.811161 7853 raft_consensus.cc:3037] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:27.812716 7853 raft_consensus.cc:504] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34123 } } peers { permanent_uuid: "8ed31e81e1864605b6c77a8123fdb062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41565 } } peers { permanent_uuid: "f4312a0972ce4ab3905b603a432eacb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } }
I20260927 00:47:27.813230 7853 leader_election.cc:288] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [CANDIDATE]: Term 1 election: Requested vote from peers 8ed31e81e1864605b6c77a8123fdb062 (127.0.98.1:41565), f4312a0972ce4ab3905b603a432eacb8 (127.0.98.3:42375)
I20260927 00:47:27.813706 7693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "722061464c204e8bac63eb066d1acc92" candidate_uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4312a0972ce4ab3905b603a432eacb8"
I20260927 00:47:27.813706 7418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "722061464c204e8bac63eb066d1acc92" candidate_uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ed31e81e1864605b6c77a8123fdb062"
I20260927 00:47:27.813846 7693 raft_consensus.cc:3037] T 722061464c204e8bac63eb066d1acc92 P f4312a0972ce4ab3905b603a432eacb8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:27.813875 7418 raft_consensus.cc:3037] T 722061464c204e8bac63eb066d1acc92 P 8ed31e81e1864605b6c77a8123fdb062 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:27.815124 7693 raft_consensus.cc:2454] T 722061464c204e8bac63eb066d1acc92 P f4312a0972ce4ab3905b603a432eacb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fedc53da95e4fc5bcdc6e2a3ee28a86 in term 1.
I20260927 00:47:27.815187 7418 raft_consensus.cc:2454] T 722061464c204e8bac63eb066d1acc92 P 8ed31e81e1864605b6c77a8123fdb062 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fedc53da95e4fc5bcdc6e2a3ee28a86 in term 1.
I20260927 00:47:27.815534 7489 leader_election.cc:302] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fedc53da95e4fc5bcdc6e2a3ee28a86, 8ed31e81e1864605b6c77a8123fdb062; no voters:
I20260927 00:47:27.815742 7853 raft_consensus.cc:2781] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:27.816015 7853 raft_consensus.cc:686] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [term 1 LEADER]: Becoming Leader. State: Replica: 1fedc53da95e4fc5bcdc6e2a3ee28a86, State: Running, Role: LEADER
I20260927 00:47:27.816453 7853 consensus_queue.cc:237] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34123 } } peers { permanent_uuid: "8ed31e81e1864605b6c77a8123fdb062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41565 } } peers { permanent_uuid: "f4312a0972ce4ab3905b603a432eacb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } }
I20260927 00:47:27.818534 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.818224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.98.2:53357)
0927 00:47:27.818374 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:27.818447 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 00:47:27.819816 7285 catalog_manager.cc:5495] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fedc53da95e4fc5bcdc6e2a3ee28a86 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fedc53da95e4fc5bcdc6e2a3ee28a86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ed31e81e1864605b6c77a8123fdb062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4312a0972ce4ab3905b603a432eacb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 } health_report { overall_health: UNKNOWN } } }
I20260927 00:47:27.843410 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.838967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40126 (local address 127.0.98.2:34123)
0927 00:47:27.839174 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:27.839178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:27.839193 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:47:27.839367 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:27.839370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:27.839444 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:27.839551 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:47:27.839818 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:27.840582 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:27.841528 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:27.841768 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:27.841857 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:27.842219 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:47:27.842902 (+ 683us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:47:27.843017 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:27.843091 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 00:47:27.843175 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 00:47:27.847177 7693 raft_consensus.cc:1261] T 722061464c204e8bac63eb066d1acc92 P f4312a0972ce4ab3905b603a432eacb8 [term 1 FOLLOWER]: Refusing update from remote peer 1fedc53da95e4fc5bcdc6e2a3ee28a86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:47:27.847604 7418 raft_consensus.cc:1261] T 722061464c204e8bac63eb066d1acc92 P 8ed31e81e1864605b6c77a8123fdb062 [term 1 FOLLOWER]: Refusing update from remote peer 1fedc53da95e4fc5bcdc6e2a3ee28a86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:47:27.847990 7855 consensus_queue.cc:1035] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4312a0972ce4ab3905b603a432eacb8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:47:27.848286 7853 consensus_queue.cc:1035] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ed31e81e1864605b6c77a8123fdb062" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:47:27.850338 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.849962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.98.3:54489)
0927 00:47:27.850169 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:27.850239 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260927 00:47:27.852342 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.851728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.98.1:34577)
0927 00:47:27.852155 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:27.852249 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333}
I20260927 00:47:27.853320 7858 mvcc.cc:205] Tried to move back new op lower bound from 7333765315975753728 to 7333765315859972096. Current Snapshot: MvccSnapshot[applied={T|T < 7333765315975753728}]
I20260927 00:47:27.856602 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.856111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.98.3:57435)
0927 00:47:27.856427 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:27.856499 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222}
I20260927 00:47:27.858508 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.858240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.98.2:34011)
0927 00:47:27.858351 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:27.858409 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 00:47:27.860970 7860 mvcc.cc:205] Tried to move back new op lower bound from 7333765315975753728 to 7333765315859972096. Current Snapshot: MvccSnapshot[applied={T|T < 7333765315975753728}]
I20260927 00:47:27.862294 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.861938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.98.1:54153)
0927 00:47:27.862087 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:27.862207 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 00:47:27.863133 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:27.862870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34945 (local address 127.0.98.2:48325)
0927 00:47:27.862996 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:27.863049 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 00:47:27.868721 7285 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:47:27.868965 7285 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:47:27.873875 7285 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3bfc9d2ca344168be147ebe68ef5d31: Sending DeleteTablet for 3 replicas of tablet 722061464c204e8bac63eb066d1acc92
I20260927 00:47:27.874766 7535 tablet_service.cc:1514] Processing DeleteTablet for tablet 722061464c204e8bac63eb066d1acc92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:27 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:40120
I20260927 00:47:27.874823 7398 tablet_service.cc:1514] Processing DeleteTablet for tablet 722061464c204e8bac63eb066d1acc92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:27 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:40770
I20260927 00:47:27.874861 7673 tablet_service.cc:1514] Processing DeleteTablet for tablet 722061464c204e8bac63eb066d1acc92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:27 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:56484
I20260927 00:47:27.875476 7868 tablet_replica.cc:330] T 722061464c204e8bac63eb066d1acc92 P 8ed31e81e1864605b6c77a8123fdb062: stopping tablet replica
I20260927 00:47:27.875686 7868 raft_consensus.cc:2229] T 722061464c204e8bac63eb066d1acc92 P 8ed31e81e1864605b6c77a8123fdb062 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:47:27.876075 7869 tablet_replica.cc:330] T 722061464c204e8bac63eb066d1acc92 P f4312a0972ce4ab3905b603a432eacb8: stopping tablet replica
I20260927 00:47:27.876341 7867 tablet_replica.cc:330] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86: stopping tablet replica
I20260927 00:47:27.876415 7869 raft_consensus.cc:2229] T 722061464c204e8bac63eb066d1acc92 P f4312a0972ce4ab3905b603a432eacb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:47:27.876562 7867 raft_consensus.cc:2229] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:47:27.876700 7869 raft_consensus.cc:2258] T 722061464c204e8bac63eb066d1acc92 P f4312a0972ce4ab3905b603a432eacb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:27.877095 7867 raft_consensus.cc:2258] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:27.877319 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 7245
I20260927 00:47:27.877343 7868 raft_consensus.cc:2258] T 722061464c204e8bac63eb066d1acc92 P 8ed31e81e1864605b6c77a8123fdb062 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:27.879251 7867 ts_tablet_manager.cc:1905] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:27.883699 7867 ts_tablet_manager.cc:1918] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:47:27.883816 7867 log.cc:1196] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-1/wal/wals/722061464c204e8bac63eb066d1acc92
I20260927 00:47:27.884140 7867 ts_tablet_manager.cc:1939] T 722061464c204e8bac63eb066d1acc92 P 1fedc53da95e4fc5bcdc6e2a3ee28a86: Deleting consensus metadata
I20260927 00:47:27.885540 7270 catalog_manager.cc:4841] TS 1fedc53da95e4fc5bcdc6e2a3ee28a86 (127.0.98.2:34123): tablet 722061464c204e8bac63eb066d1acc92 (table test-table [id=5e75f99151e446cabafd6abd56b0619d]) successfully deleted
I20260927 00:47:27.888861 7869 ts_tablet_manager.cc:1905] T 722061464c204e8bac63eb066d1acc92 P f4312a0972ce4ab3905b603a432eacb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:27.891376 7869 ts_tablet_manager.cc:1918] T 722061464c204e8bac63eb066d1acc92 P f4312a0972ce4ab3905b603a432eacb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:47:27.891499 7869 log.cc:1196] T 722061464c204e8bac63eb066d1acc92 P f4312a0972ce4ab3905b603a432eacb8: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790470006085796-392-0/minicluster-data/ts-2/wal/wals/722061464c204e8bac63eb066d1acc92
W20260927 00:47:27.892102 7843 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:27.888762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41565 (local address 127.0.0.1:40782)
0927 00:47:27.888920 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:27.888939 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:27.889055 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:27.891947 (+ 2892us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41565: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 00:47:27.892217 7869 ts_tablet_manager.cc:1939] T 722061464c204e8bac63eb066d1acc92 P f4312a0972ce4ab3905b603a432eacb8: Deleting consensus metadata
I20260927 00:47:27.892290 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 7472
W20260927 00:47:27.893330 7270 catalog_manager.cc:4584] TS 8ed31e81e1864605b6c77a8123fdb062 (127.0.98.1:41565): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 722061464c204e8bac63eb066d1acc92: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41565: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 00:47:27.894675 7271 catalog_manager.cc:4841] TS f4312a0972ce4ab3905b603a432eacb8 (127.0.98.3:42375): tablet 722061464c204e8bac63eb066d1acc92 (table test-table [id=5e75f99151e446cabafd6abd56b0619d]) successfully deleted
I20260927 00:47:27.899626 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 7608
I20260927 00:47:27.906620 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 6964
I20260927 00:47:27.913771 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 7065
2026-09-27T00:47:27Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7462 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:47:27 dist-test-slave-86rs krb5kdc[7872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:47:27 dist-test-slave-86rs krb5kdc[7872](info): set up 2 sockets
Sep 27 00:47:27 dist-test-slave-86rs krb5kdc[7872](info): commencing operation
krb5kdc: starting...
W20260927 00:47:28.973318 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:47:29 dist-test-slave-86rs krb5kdc[7872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:47:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:47:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 00:47:29.120711 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46561
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:29.258164 7885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:29.258561 7885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:29.258610 7885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:29.263163 7885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:47:29.263271 7885 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 00:47:29.263290 7885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:29.263310 7885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:29.263327 7885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:47:29.263343 7885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:47:29.269599 7885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46561
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:29.271149 7885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:29.272598 7885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:29.283463 7891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:29.283537 7885 server_base.cc:1034] running on GCE node
W20260927 00:47:29.283463 7890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:29.283483 7893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:29.284600 7885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:29.286185 7885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:29.287359 7885 hybrid_clock.cc:648] HybridClock initialized: now 1790470049287326 us; error 65 us; skew 500 ppm
Sep 27 00:47:29 dist-test-slave-86rs krb5kdc[7872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470049, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:29.290907 7885 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 00:47:29.292291 7885 webserver.cc:460] Webserver started at http://127.0.98.62:41947/ using document root <none> and password file <none>
I20260927 00:47:29.293049 7885 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:29.293164 7885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:29.293467 7885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:29.296401 7885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "c7dec1acba204282bf94e1473f15c1bb"
format_stamp: "Formatted at 2026-09-27 00:47:29 on dist-test-slave-86rs"
server_key: "59bff64eb102576839be1e5871fb7d5d"
server_key_iv: "c51274c5d270de214fac18a0090dafb3"
server_key_version: "encryptionkey@0"
I20260927 00:47:29.297225 7885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "c7dec1acba204282bf94e1473f15c1bb"
format_stamp: "Formatted at 2026-09-27 00:47:29 on dist-test-slave-86rs"
server_key: "59bff64eb102576839be1e5871fb7d5d"
server_key_iv: "c51274c5d270de214fac18a0090dafb3"
server_key_version: "encryptionkey@0"
I20260927 00:47:29.302589 7885 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 00:47:29.306113 7900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:29.307967 7885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:47:29.308207 7885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "c7dec1acba204282bf94e1473f15c1bb"
format_stamp: "Formatted at 2026-09-27 00:47:29 on dist-test-slave-86rs"
server_key: "59bff64eb102576839be1e5871fb7d5d"
server_key_iv: "c51274c5d270de214fac18a0090dafb3"
server_key_version: "encryptionkey@0"
I20260927 00:47:29.308463 7885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:29.331044 7885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:29.335692 7885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:29.335970 7885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:29.346267 7885 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46561
I20260927 00:47:29.346266 7962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46561 every 8 connection(s)
I20260927 00:47:29.347654 7885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:47:29.350256 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 7885
I20260927 00:47:29.350443 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:47:29.350847 392 external_mini_cluster.cc:1428] Setting key 7395dc649b287d42139434725bd15777
I20260927 00:47:29.351872 7963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:29.360193 7963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7dec1acba204282bf94e1473f15c1bb: Bootstrap starting.
Sep 27 00:47:29 dist-test-slave-86rs krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:47:29.363701 7963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7dec1acba204282bf94e1473f15c1bb: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:29.365324 7963 log.cc:824] T 00000000000000000000000000000000 P c7dec1acba204282bf94e1473f15c1bb: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:29.368491 7963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7dec1acba204282bf94e1473f15c1bb: No bootstrap required, opened a new log
I20260927 00:47:29.369213 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:29.352729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38238 (local address 127.0.98.62:46561)
0927 00:47:29.353272 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:29.353295 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:29.353349 (+ 54us) server_negotiation.cc:407] Connection header received
0927 00:47:29.354347 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:29.354374 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:29.354916 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:29.355383 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:29.356155 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:29.357756 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:29.358724 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:29.359455 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:29.359629 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:29.362841 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:29.362848 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:29.362869 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:29.362907 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:29.365301 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:29.365889 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:29.365900 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:29.365907 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:29.366043 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:29.366376 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:29.366378 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:29.366379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:29.367034 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:29.367256 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:29.367576 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 00:47:29.367963 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":179,"threads_started":1}
I20260927 00:47:29.372753 7963 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7dec1acba204282bf94e1473f15c1bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7dec1acba204282bf94e1473f15c1bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46561 } }
I20260927 00:47:29.373025 7963 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7dec1acba204282bf94e1473f15c1bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:29.373062 7963 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7dec1acba204282bf94e1473f15c1bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7dec1acba204282bf94e1473f15c1bb, State: Initialized, Role: FOLLOWER
I20260927 00:47:29.373589 7963 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7dec1acba204282bf94e1473f15c1bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7dec1acba204282bf94e1473f15c1bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46561 } }
I20260927 00:47:29.373737 7963 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7dec1acba204282bf94e1473f15c1bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:47:29.373790 7963 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7dec1acba204282bf94e1473f15c1bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:47:29.373873 7963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7dec1acba204282bf94e1473f15c1bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:29.375133 7963 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7dec1acba204282bf94e1473f15c1bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7dec1acba204282bf94e1473f15c1bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46561 } }
I20260927 00:47:29.375576 7963 leader_election.cc:302] T 00000000000000000000000000000000 P c7dec1acba204282bf94e1473f15c1bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7dec1acba204282bf94e1473f15c1bb; no voters:
I20260927 00:47:29.375981 7963 leader_election.cc:288] T 00000000000000000000000000000000 P c7dec1acba204282bf94e1473f15c1bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:47:29.376139 7968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7dec1acba204282bf94e1473f15c1bb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:29.376461 7968 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7dec1acba204282bf94e1473f15c1bb [term 1 LEADER]: Becoming Leader. State: Replica: c7dec1acba204282bf94e1473f15c1bb, State: Running, Role: LEADER
I20260927 00:47:29.376895 7968 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7dec1acba204282bf94e1473f15c1bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7dec1acba204282bf94e1473f15c1bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46561 } }
I20260927 00:47:29.377609 7963 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7dec1acba204282bf94e1473f15c1bb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:47:29.378995 7970 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7dec1acba204282bf94e1473f15c1bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7dec1acba204282bf94e1473f15c1bb. Latest consensus state: current_term: 1 leader_uuid: "c7dec1acba204282bf94e1473f15c1bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7dec1acba204282bf94e1473f15c1bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46561 } } }
I20260927 00:47:29.379142 7970 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7dec1acba204282bf94e1473f15c1bb [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:29.379431 7969 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7dec1acba204282bf94e1473f15c1bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7dec1acba204282bf94e1473f15c1bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7dec1acba204282bf94e1473f15c1bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46561 } } }
I20260927 00:47:29.379536 7969 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7dec1acba204282bf94e1473f15c1bb [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:29.379556 7977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:47:29.383237 7977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:47:29.390959 7977 catalog_manager.cc:1347] Generated new cluster ID: 821680d62af140d587ca0acc2c604a67
I20260927 00:47:29.391103 7977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:47:29.398164 7977 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:47:29.399298 7977 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:47:29.406003 7977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7dec1acba204282bf94e1473f15c1bb: Generated new TSK 0
I20260927 00:47:29.407156 7977 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 00:47:29.450047 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46561
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:29.592527 7989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:29.592931 7989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:29.592986 7989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:29.597834 7989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:29.597944 7989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:29.597994 7989 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:47:29.598057 7989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:47:29.604720 7989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:29.606380 7989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:29.607898 7989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:29.620692 7994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:29.620692 7997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:29.620692 7995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:29.621196 7989 server_base.cc:1034] running on GCE node
I20260927 00:47:29.621699 7989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:29.622510 7989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:29.623729 7989 hybrid_clock.cc:648] HybridClock initialized: now 1790470049623700 us; error 46 us; skew 500 ppm
Sep 27 00:47:29 dist-test-slave-86rs krb5kdc[7872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470049, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:29.627260 7989 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 00:47:29.628679 7989 webserver.cc:460] Webserver started at http://127.0.98.1:46011/ using document root <none> and password file <none>
I20260927 00:47:29.629438 7989 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:29.629520 7989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:29.629755 7989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:29.632654 7989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-0/data/instance:
uuid: "09c44699d7f34db688c84b5d2e338142"
format_stamp: "Formatted at 2026-09-27 00:47:29 on dist-test-slave-86rs"
server_key: "cafbe3970959fcbe329db1818cce5e37"
server_key_iv: "b86f8e3e9b500bfb9ce67e7f8235c17b"
server_key_version: "encryptionkey@0"
I20260927 00:47:29.633392 7989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-0/wal/instance:
uuid: "09c44699d7f34db688c84b5d2e338142"
format_stamp: "Formatted at 2026-09-27 00:47:29 on dist-test-slave-86rs"
server_key: "cafbe3970959fcbe329db1818cce5e37"
server_key_iv: "b86f8e3e9b500bfb9ce67e7f8235c17b"
server_key_version: "encryptionkey@0"
I20260927 00:47:29.638649 7989 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 00:47:29.642174 8004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:29.643932 7989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:47:29.644081 7989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "09c44699d7f34db688c84b5d2e338142"
format_stamp: "Formatted at 2026-09-27 00:47:29 on dist-test-slave-86rs"
server_key: "cafbe3970959fcbe329db1818cce5e37"
server_key_iv: "b86f8e3e9b500bfb9ce67e7f8235c17b"
server_key_version: "encryptionkey@0"
I20260927 00:47:29.644282 7989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:29.668548 7989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:29.673070 7989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:29.673285 7989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:29.675172 7989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:29.675289 7989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:29.675369 7989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:29.675417 7989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:47:29 dist-test-slave-86rs krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470049, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:47:29.689121 7989 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40447
I20260927 00:47:29.689168 8119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40447 every 8 connection(s)
I20260927 00:47:29.690344 7989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:47:29.693048 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:29.676455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60489 (local address 127.0.98.62:46561)
0927 00:47:29.676618 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:29.676622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:29.678222 (+ 1600us) server_negotiation.cc:407] Connection header received
0927 00:47:29.679882 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:29.679887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:29.679961 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:29.680031 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:29.682058 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:29.683070 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:29.684754 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:29.685030 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:29.685175 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:29.688759 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:29.688764 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:29.688767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:29.688806 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:29.690616 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:29.691285 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:29.691289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:29.691291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:29.691389 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:29.691773 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:29.691776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:29.691777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:29.692001 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:29.692083 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:29.692760 (+ 677us) server_negotiation.cc:299] Negotiation successful
0927 00:47:29.692900 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:47:29.694101 8013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:29.677305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46561 (local address 127.0.98.1:60489)
0927 00:47:29.678005 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:29.678063 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:29.679530 (+ 1467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:29.680578 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:29.680587 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:29.681177 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:29.681874 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:29.681895 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:29.683259 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:29.683264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:29.684596 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:29.684616 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:29.685256 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:29.685260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:29.685468 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:29.686241 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:29.686262 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:29.688492 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:29.690745 (+ 2253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:29.690751 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:29.690770 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:29.691153 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:29.691511 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:29.691514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:29.691516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:29.691668 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:29.692096 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:29.692102 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:29.692486 (+ 384us) client_negotiation.cc:769] Sending connection context
0927 00:47:29.692705 (+ 219us) client_negotiation.cc:240] Negotiation successful
0927 00:47:29.693042 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":188,"threads_started":1}
I20260927 00:47:29.695783 8120 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46561
I20260927 00:47:29.696058 8120 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:29.697290 8120 heartbeater.cc:511] Master 127.0.98.62:46561 requested a full tablet report, sending...
I20260927 00:47:29.699576 7914 ts_manager.cc:194] Registered new tserver with Master: 09c44699d7f34db688c84b5d2e338142 (127.0.98.1:40447)
I20260927 00:47:29.700006 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 7989
I20260927 00:47:29.700177 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:47:29.700627 392 external_mini_cluster.cc:1428] Setting key e0d1c9bd2373d69418b79baba6e4741d
I20260927 00:47:29.701231 7914 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60489
I20260927 00:47:29.707579 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:29.698640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58697 (local address 127.0.98.62:46561)
0927 00:47:29.698793 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:29.698798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:29.698960 (+ 162us) server_negotiation.cc:407] Connection header received
0927 00:47:29.699098 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:29.699101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:29.699150 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:29.699241 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:29.700354 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:29.701143 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:29.702405 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:29.702734 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:29.702858 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:29.704369 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:29.704374 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:29.704377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:29.704405 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:29.705840 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:29.706375 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:29.706378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:29.706380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:29.706452 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:29.706750 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:29.706752 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:29.706753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:29.706924 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:29.707003 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:29.707332 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 00:47:29.707442 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:47:29.707753 8126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:29.698502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46561 (local address 127.0.98.1:58697)
0927 00:47:29.698870 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:29.698884 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:29.698983 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:29.699268 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:29.699270 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:29.699481 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:29.699601 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:29.699607 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:29.701315 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:29.701318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:29.702232 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:29.702243 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:29.702866 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:29.702870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:29.702972 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:29.703652 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:29.703667 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:29.704183 (+ 516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:29.705952 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:29.705959 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:29.705964 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:29.706275 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:29.706543 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:29.706549 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:29.706554 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:29.706662 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:29.707025 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:29.707030 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:29.707154 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 00:47:29.707283 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 00:47:29.707481 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 00:47:29.732510 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46561
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:29.872270 8129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:29.872707 8129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:29.872810 8129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:29.877209 8129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:29.877315 8129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:29.877393 8129 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:47:29.877480 8129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:47:29.884478 8129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:29.886080 8129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:29.887529 8129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:29.900022 8135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:29.900005 8134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:29.900246 8129 server_base.cc:1034] running on GCE node
W20260927 00:47:29.900005 8137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:29.900981 8129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:29.901826 8129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:29.903009 8129 hybrid_clock.cc:648] HybridClock initialized: now 1790470049902983 us; error 42 us; skew 500 ppm
Sep 27 00:47:29 dist-test-slave-86rs krb5kdc[7872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470049, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:29.906637 8129 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 00:47:29.907869 8129 webserver.cc:460] Webserver started at http://127.0.98.2:38239/ using document root <none> and password file <none>
I20260927 00:47:29.908697 8129 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:29.908802 8129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:29.909078 8129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:29.911674 8129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-1/data/instance:
uuid: "a0ae7faedd48405bbc5255ae0acbc6e2"
format_stamp: "Formatted at 2026-09-27 00:47:29 on dist-test-slave-86rs"
server_key: "a517b135d43d71c746aadec37681a60f"
server_key_iv: "7cd930fda60fc78c67a1c726831ad90b"
server_key_version: "encryptionkey@0"
I20260927 00:47:29.912452 8129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a0ae7faedd48405bbc5255ae0acbc6e2"
format_stamp: "Formatted at 2026-09-27 00:47:29 on dist-test-slave-86rs"
server_key: "a517b135d43d71c746aadec37681a60f"
server_key_iv: "7cd930fda60fc78c67a1c726831ad90b"
server_key_version: "encryptionkey@0"
I20260927 00:47:29.917207 8129 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:47:29.920331 8144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:29.922065 8129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:47:29.922248 8129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "a0ae7faedd48405bbc5255ae0acbc6e2"
format_stamp: "Formatted at 2026-09-27 00:47:29 on dist-test-slave-86rs"
server_key: "a517b135d43d71c746aadec37681a60f"
server_key_iv: "7cd930fda60fc78c67a1c726831ad90b"
server_key_version: "encryptionkey@0"
I20260927 00:47:29.922397 8129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:29.950543 8129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:29.954545 8129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:29.954809 8129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:29.957196 8129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:29.957293 8129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:29.957362 8129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:29.957397 8129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:47:29 dist-test-slave-86rs krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470049, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:47:29.971566 8129 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35833
I20260927 00:47:29.971638 8259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35833 every 8 connection(s)
I20260927 00:47:29.972844 8129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:47:29.974057 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:29.957979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52713 (local address 127.0.98.62:46561)
0927 00:47:29.958166 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:29.958171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:29.959821 (+ 1650us) server_negotiation.cc:407] Connection header received
0927 00:47:29.961521 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:29.961526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:29.961608 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:29.961722 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:29.964147 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:29.964846 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:29.966216 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:29.966468 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:29.966603 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:29.969707 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:29.969712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:29.969714 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:29.969747 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:29.971525 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:29.972143 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:29.972148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:29.972149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:29.972279 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:29.972649 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:29.972653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:29.972655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:29.972916 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:29.973028 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:29.973695 (+ 667us) server_negotiation.cc:299] Negotiation successful
0927 00:47:29.973865 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 00:47:29.975086 8153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:29.958861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46561 (local address 127.0.98.2:52713)
0927 00:47:29.959594 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:29.959654 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:29.961151 (+ 1497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:29.962078 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:29.962092 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:29.962795 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:29.963902 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:29.963932 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:29.965008 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:29.965012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:29.966073 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:29.966083 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:29.966674 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:29.966678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:29.966897 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:29.967654 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:29.967674 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:29.969479 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:29.971684 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:29.971691 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:29.971711 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:29.972020 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:29.972362 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:29.972365 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:29.972367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:29.972542 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:29.973023 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:29.973031 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:29.973351 (+ 320us) client_negotiation.cc:769] Sending connection context
0927 00:47:29.973621 (+ 270us) client_negotiation.cc:240] Negotiation successful
0927 00:47:29.974034 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":204,"threads_started":1}
I20260927 00:47:29.976989 8260 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46561
I20260927 00:47:29.977236 8260 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:29.978125 8260 heartbeater.cc:511] Master 127.0.98.62:46561 requested a full tablet report, sending...
I20260927 00:47:29.979990 7917 ts_manager.cc:194] Registered new tserver with Master: a0ae7faedd48405bbc5255ae0acbc6e2 (127.0.98.2:35833)
I20260927 00:47:29.981055 7917 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52713
I20260927 00:47:29.981838 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 8129
I20260927 00:47:29.981997 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:47:29.982396 392 external_mini_cluster.cc:1428] Setting key 8f3d9b1ffe175bed6c80f4e95cab8c25
I20260927 00:47:29.989490 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:29.980487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41585 (local address 127.0.98.62:46561)
0927 00:47:29.980654 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:29.980660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:29.980913 (+ 253us) server_negotiation.cc:407] Connection header received
0927 00:47:29.981030 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:29.981033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:29.981085 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:29.981191 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:29.981918 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:29.982476 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:29.983548 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:29.983872 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:29.984011 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:29.985601 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:29.985613 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:29.985615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:29.985638 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:29.987400 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:29.987922 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:29.987925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:29.987933 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:29.988062 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:29.988446 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:29.988449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:29.988473 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:29.988721 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:29.988839 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:29.989147 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 00:47:29.989273 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 00:47:29.989512 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:29.980412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46561 (local address 127.0.98.2:41585)
0927 00:47:29.980788 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:29.980804 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:29.980931 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:29.981263 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:29.981266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:29.981573 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:29.981734 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:29.981744 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:29.982622 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:29.982626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:29.983417 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:29.983427 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:29.984191 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:29.984194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:29.984334 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:29.985031 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:29.985044 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:29.985437 (+ 393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:29.987545 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:29.987548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:29.987550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:29.987806 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:29.988169 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:29.988174 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:29.988176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:29.988332 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:29.988824 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:29.988827 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:29.988939 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 00:47:29.989090 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 00:47:29.989274 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":168,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 00:47:30.016781 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46561
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:30.158154 8269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:30.158595 8269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:30.158660 8269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:30.163560 8269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:30.163702 8269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:30.163749 8269 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:47:30.163789 8269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:47:30.170722 8269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:30.172436 8269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:30.173905 8269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:47:30.187348 8269 server_base.cc:1034] running on GCE node
W20260927 00:47:30.187350 8274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:30.187384 8277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:30.187352 8275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:30.188540 8269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:30.189519 8269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:30.190745 8269 hybrid_clock.cc:648] HybridClock initialized: now 1790470050190697 us; error 68 us; skew 500 ppm
Sep 27 00:47:30 dist-test-slave-86rs krb5kdc[7872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470050, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:30.194331 8269 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 00:47:30.195670 8269 webserver.cc:460] Webserver started at http://127.0.98.3:45421/ using document root <none> and password file <none>
I20260927 00:47:30.196574 8269 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:30.196646 8269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:30.196877 8269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:30.199451 8269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-2/data/instance:
uuid: "2f4a4619b3e04e6e8e99a20b640d4650"
format_stamp: "Formatted at 2026-09-27 00:47:30 on dist-test-slave-86rs"
server_key: "c681f202d2b31ccb3b94c51676fa30bd"
server_key_iv: "ed42b395acbf307705c02295aa439a4b"
server_key_version: "encryptionkey@0"
I20260927 00:47:30.200124 8269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2f4a4619b3e04e6e8e99a20b640d4650"
format_stamp: "Formatted at 2026-09-27 00:47:30 on dist-test-slave-86rs"
server_key: "c681f202d2b31ccb3b94c51676fa30bd"
server_key_iv: "ed42b395acbf307705c02295aa439a4b"
server_key_version: "encryptionkey@0"
I20260927 00:47:30.205083 8269 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 00:47:30.208220 8284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:30.209897 8269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:47:30.210067 8269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "2f4a4619b3e04e6e8e99a20b640d4650"
format_stamp: "Formatted at 2026-09-27 00:47:30 on dist-test-slave-86rs"
server_key: "c681f202d2b31ccb3b94c51676fa30bd"
server_key_iv: "ed42b395acbf307705c02295aa439a4b"
server_key_version: "encryptionkey@0"
I20260927 00:47:30.210239 8269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:30.226379 8269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:30.230444 8269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:30.230696 8269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:30.232555 8269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:30.232637 8269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:30.232720 8269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:30.232764 8269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:47:30 dist-test-slave-86rs krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470050, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:47:30.245805 8269 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42585
I20260927 00:47:30.245860 8399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42585 every 8 connection(s)
I20260927 00:47:30.247000 8269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:47:30.250407 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:30.233652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32861 (local address 127.0.98.62:46561)
0927 00:47:30.233860 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:30.233868 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:30.234921 (+ 1053us) server_negotiation.cc:407] Connection header received
0927 00:47:30.236309 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:30.236313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:30.236398 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:30.236505 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:30.238138 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.238889 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.240810 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.241194 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.241276 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:30.245764 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:30.245770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:30.245773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:30.245814 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:30.247921 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:30.248600 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:30.248607 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:30.248635 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:30.248735 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:30.249106 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:30.249109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:30.249110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:30.249332 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:30.249414 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:30.250112 (+ 698us) server_negotiation.cc:299] Negotiation successful
0927 00:47:30.250263 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 00:47:30.251616 8293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:30.234159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46561 (local address 127.0.98.3:32861)
0927 00:47:30.234718 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:30.234778 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:30.235985 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:30.236851 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:30.236860 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:30.237357 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:30.237972 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:30.237988 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.239170 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.239174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:30.240604 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:30.240618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.241532 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.241539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:30.241844 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:30.242634 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:30.242668 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:30.245443 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:30.248062 (+ 2619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:30.248072 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:30.248092 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:30.248466 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:30.248849 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:30.248852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:30.248853 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:30.249006 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:30.249428 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:30.249434 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:30.249793 (+ 359us) client_negotiation.cc:769] Sending connection context
0927 00:47:30.250021 (+ 228us) client_negotiation.cc:240] Negotiation successful
0927 00:47:30.250478 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":157,"threads_started":1}
I20260927 00:47:30.253353 8400 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46561
I20260927 00:47:30.253522 8400 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:30.254201 8400 heartbeater.cc:511] Master 127.0.98.62:46561 requested a full tablet report, sending...
I20260927 00:47:30.255555 7917 ts_manager.cc:194] Registered new tserver with Master: 2f4a4619b3e04e6e8e99a20b640d4650 (127.0.98.3:42585)
I20260927 00:47:30.256374 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 8269
I20260927 00:47:30.256533 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:47:30.256526 7917 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:32861
I20260927 00:47:30.256963 392 external_mini_cluster.cc:1428] Setting key ecabd828f89936e111beef3c5cd01a97
I20260927 00:47:30.259331 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:47:30.266144 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:30.255971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46561 (local address 127.0.98.3:42189)
0927 00:47:30.256414 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:30.256431 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:30.256540 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:30.257645 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:30.257781 (+ 136us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:30.258063 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:30.258259 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:30.258269 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.259093 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.259097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:30.259807 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:30.259817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.260353 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.260360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:30.260496 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:30.261151 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:30.261172 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:30.261742 (+ 570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:30.264153 (+ 2411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:30.264157 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:30.264159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:30.264568 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:30.264886 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:30.264888 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:30.264889 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:30.264965 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:30.265463 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:30.265465 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:30.265614 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 00:47:30.265743 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 00:47:30.265900 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":1,"thread_start_us":184,"threads_started":1}
I20260927 00:47:30.266203 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:30.256071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42189 (local address 127.0.98.62:46561)
0927 00:47:30.257412 (+ 1341us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:30.257416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:30.257427 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:47:30.257464 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:30.257466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:30.257525 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:30.257637 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:30.258379 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.258946 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.259952 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.260199 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.260347 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:30.261953 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:30.261959 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:30.261964 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:30.262001 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:30.263986 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:30.264673 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:30.264680 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:30.264685 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:30.264789 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:30.265056 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:30.265061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:30.265066 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:30.265365 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:30.265481 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:30.265832 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 00:47:30.265992 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1264}
Sep 27 00:47:30 dist-test-slave-86rs krb5kdc[7872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470050, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:47:30 dist-test-slave-86rs krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:30 dist-test-slave-86rs krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 00:47:30.291237 7966 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:30.283670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38252 (local address 127.0.98.62:46561)
0927 00:47:30.283830 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:30.283834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:30.283850 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:47:30.283905 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:30.283907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:30.283957 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:30.284055 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:30.284973 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.285587 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.286522 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.286750 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.286837 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:30.291073 (+ 4236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38252: BlockingRecv error: recv got EOF from 127.0.0.1:38252 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 00:47:30 dist-test-slave-86rs krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470050, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:47:30.303005 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:30.293636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38256 (local address 127.0.98.62:46561)
0927 00:47:30.293798 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:30.293802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:30.293896 (+ 94us) server_negotiation.cc:407] Connection header received
0927 00:47:30.294040 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:30.294043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:30.294096 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:30.294176 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:30.294822 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.295469 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.296490 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.296754 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.296839 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:30.299502 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:30.299506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:30.299508 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:30.299540 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:30.301384 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:30.301788 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:30.301791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:30.301793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:30.301883 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:30.302116 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:30.302118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:30.302119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:30.302323 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:30.302438 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:30.302636 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 00:47:30.302748 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 00:47:30.306932 7917 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:47:30.310802 7917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:47:30.328862 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:30.321749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42585 (local address 127.0.0.1:42700)
0927 00:47:30.322683 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:30.322752 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:30.322905 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:30.323922 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:30.323929 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:30.323954 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:30.324151 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:30.324158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.325686 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.325689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:30.327145 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:30.327158 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.328129 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.328133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:30.328281 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:30.328330 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 00:47:30.328464 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 00:47:30.328648 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":1,"thread_start_us":131,"threads_started":1}
I20260927 00:47:30.328863 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:30.322162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35833 (local address 127.0.0.1:48412)
0927 00:47:30.323022 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:30.323037 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:30.323132 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:30.324061 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:30.324064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:30.324082 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:30.324250 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:30.324257 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.325424 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.325430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:30.327193 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:30.327202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.328431 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.328438 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:30.328548 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:30.328570 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:47:30.328632 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 00:47:30.328691 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"mutex_wait_us":99,"thread_start_us":95,"threads_started":1}
I20260927 00:47:30.329300 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:30.322726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40447 (local address 127.0.0.1:34832)
0927 00:47:30.323494 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:30.323507 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:30.323626 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:30.324244 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:30.324246 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:30.324261 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:30.324364 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:30.324370 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.325900 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.325904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:30.327245 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:30.327254 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.328650 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.328653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:30.328749 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:30.328772 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:47:30.329045 (+ 273us) client_negotiation.cc:240] Negotiation successful
0927 00:47:30.329147 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":79,"thread_start_us":99,"threads_started":1}
I20260927 00:47:30.329622 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:30.321928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42700 (local address 127.0.98.3:42585)
0927 00:47:30.323281 (+ 1353us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:30.323298 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:30.323327 (+ 29us) server_negotiation.cc:407] Connection header received
0927 00:47:30.323427 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:30.323434 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:30.323751 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:30.323943 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:30.324297 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.325559 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.327284 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.327940 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.328075 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:30.328359 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:30.329284 (+ 925us) server_negotiation.cc:299] Negotiation successful
0927 00:47:30.329430 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1215,"thread_start_us":200,"threads_started":1}
I20260927 00:47:30.329718 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:30.322392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48412 (local address 127.0.98.2:35833)
0927 00:47:30.323557 (+ 1165us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:30.323600 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:30.323621 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:47:30.323708 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:30.323713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:30.323898 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:30.324414 (+ 516us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:30.324567 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.325309 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.327331 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.328273 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.328425 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:30.328616 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:30.328676 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 00:47:30.328801 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1074,"thread_start_us":118,"threads_started":1}
I20260927 00:47:30.329794 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:30.322963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34832 (local address 127.0.98.1:40447)
0927 00:47:30.323371 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:30.323376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:30.323620 (+ 244us) server_negotiation.cc:407] Connection header received
0927 00:47:30.323730 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:30.323734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:30.323897 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:30.324056 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:30.324486 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.325779 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.327372 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.328320 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.328425 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:30.328582 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:30.329480 (+ 898us) server_negotiation.cc:299] Negotiation successful
0927 00:47:30.329610 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":117,"threads_started":1}
I20260927 00:47:30.333586 8334 tablet_service.cc:1467] Processing CreateTablet for tablet cda3caa885244130b0a7358f0d545a94 (DEFAULT_TABLE table=test-table [id=af6ff38eb9a745acbd8036ae883ee646]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:30.334447 8054 tablet_service.cc:1467] Processing CreateTablet for tablet cda3caa885244130b0a7358f0d545a94 (DEFAULT_TABLE table=test-table [id=af6ff38eb9a745acbd8036ae883ee646]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:30.334441 8194 tablet_service.cc:1467] Processing CreateTablet for tablet cda3caa885244130b0a7358f0d545a94 (DEFAULT_TABLE table=test-table [id=af6ff38eb9a745acbd8036ae883ee646]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:30.335188 8334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cda3caa885244130b0a7358f0d545a94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:30.335564 8054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cda3caa885244130b0a7358f0d545a94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:30.335566 8194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cda3caa885244130b0a7358f0d545a94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:30.344187 8428 tablet_bootstrap.cc:492] T cda3caa885244130b0a7358f0d545a94 P 09c44699d7f34db688c84b5d2e338142: Bootstrap starting.
I20260927 00:47:30.346753 8430 tablet_bootstrap.cc:492] T cda3caa885244130b0a7358f0d545a94 P 2f4a4619b3e04e6e8e99a20b640d4650: Bootstrap starting.
I20260927 00:47:30.346977 8429 tablet_bootstrap.cc:492] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2: Bootstrap starting.
I20260927 00:47:30.348089 8428 tablet_bootstrap.cc:654] T cda3caa885244130b0a7358f0d545a94 P 09c44699d7f34db688c84b5d2e338142: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:30.349139 8428 log.cc:824] T cda3caa885244130b0a7358f0d545a94 P 09c44699d7f34db688c84b5d2e338142: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:30.351131 8430 tablet_bootstrap.cc:654] T cda3caa885244130b0a7358f0d545a94 P 2f4a4619b3e04e6e8e99a20b640d4650: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:30.351680 8428 tablet_bootstrap.cc:492] T cda3caa885244130b0a7358f0d545a94 P 09c44699d7f34db688c84b5d2e338142: No bootstrap required, opened a new log
I20260927 00:47:30.351936 8428 ts_tablet_manager.cc:1397] T cda3caa885244130b0a7358f0d545a94 P 09c44699d7f34db688c84b5d2e338142: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260927 00:47:30.352213 8430 log.cc:824] T cda3caa885244130b0a7358f0d545a94 P 2f4a4619b3e04e6e8e99a20b640d4650: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:30.354000 8429 tablet_bootstrap.cc:654] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:30.355023 8429 log.cc:824] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:30.356632 8430 tablet_bootstrap.cc:492] T cda3caa885244130b0a7358f0d545a94 P 2f4a4619b3e04e6e8e99a20b640d4650: No bootstrap required, opened a new log
I20260927 00:47:30.356891 8430 ts_tablet_manager.cc:1397] T cda3caa885244130b0a7358f0d545a94 P 2f4a4619b3e04e6e8e99a20b640d4650: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 00:47:30.357127 8428 raft_consensus.cc:348] T cda3caa885244130b0a7358f0d545a94 P 09c44699d7f34db688c84b5d2e338142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } } peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } }
I20260927 00:47:30.357455 8428 raft_consensus.cc:374] T cda3caa885244130b0a7358f0d545a94 P 09c44699d7f34db688c84b5d2e338142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:30.357509 8428 raft_consensus.cc:729] T cda3caa885244130b0a7358f0d545a94 P 09c44699d7f34db688c84b5d2e338142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c44699d7f34db688c84b5d2e338142, State: Initialized, Role: FOLLOWER
I20260927 00:47:30.358167 8429 tablet_bootstrap.cc:492] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2: No bootstrap required, opened a new log
I20260927 00:47:30.358134 8428 consensus_queue.cc:260] T cda3caa885244130b0a7358f0d545a94 P 09c44699d7f34db688c84b5d2e338142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } } peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } }
I20260927 00:47:30.358357 8429 ts_tablet_manager.cc:1397] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260927 00:47:30.359318 8428 ts_tablet_manager.cc:1428] T cda3caa885244130b0a7358f0d545a94 P 09c44699d7f34db688c84b5d2e338142: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260927 00:47:30.359813 8120 heartbeater.cc:503] Master 127.0.98.62:46561 was elected leader, sending a full tablet report...
I20260927 00:47:30.362569 8430 raft_consensus.cc:348] T cda3caa885244130b0a7358f0d545a94 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } } peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } }
I20260927 00:47:30.363008 8430 raft_consensus.cc:374] T cda3caa885244130b0a7358f0d545a94 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:30.363106 8430 raft_consensus.cc:729] T cda3caa885244130b0a7358f0d545a94 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f4a4619b3e04e6e8e99a20b640d4650, State: Initialized, Role: FOLLOWER
I20260927 00:47:30.363770 8430 consensus_queue.cc:260] T cda3caa885244130b0a7358f0d545a94 P 2f4a4619b3e04e6e8e99a20b640d4650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } } peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } }
I20260927 00:47:30.363858 8429 raft_consensus.cc:348] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } } peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } }
I20260927 00:47:30.364207 8429 raft_consensus.cc:374] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:30.364375 8429 raft_consensus.cc:729] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0ae7faedd48405bbc5255ae0acbc6e2, State: Initialized, Role: FOLLOWER
I20260927 00:47:30.364892 8429 consensus_queue.cc:260] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } } peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } }
I20260927 00:47:30.365231 8430 ts_tablet_manager.cc:1428] T cda3caa885244130b0a7358f0d545a94 P 2f4a4619b3e04e6e8e99a20b640d4650: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260927 00:47:30.365657 8400 heartbeater.cc:503] Master 127.0.98.62:46561 was elected leader, sending a full tablet report...
I20260927 00:47:30.365768 8260 heartbeater.cc:503] Master 127.0.98.62:46561 was elected leader, sending a full tablet report...
I20260927 00:47:30.367393 8429 ts_tablet_manager.cc:1428] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2: Time spent starting tablet: real 0.009s user 0.003s sys 0.004s
W20260927 00:47:30.442154 8121 tablet.cc:2369] T cda3caa885244130b0a7358f0d545a94 P 09c44699d7f34db688c84b5d2e338142: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:47:30.474679 8261 tablet.cc:2369] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:47:30.498242 8401 tablet.cc:2369] T cda3caa885244130b0a7358f0d545a94 P 2f4a4619b3e04e6e8e99a20b640d4650: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:47:30.499491 8436 raft_consensus.cc:482] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:47:30.499758 8436 raft_consensus.cc:504] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } } peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } }
I20260927 00:47:30.501338 8436 leader_election.cc:288] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f4a4619b3e04e6e8e99a20b640d4650 (127.0.98.3:42585), 09c44699d7f34db688c84b5d2e338142 (127.0.98.1:40447)
I20260927 00:47:30.507042 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:30.502370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34367 (local address 127.0.98.1:40447)
0927 00:47:30.502516 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:30.502521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:30.502566 (+ 45us) server_negotiation.cc:407] Connection header received
0927 00:47:30.502743 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:30.502746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:30.502811 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:30.502898 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:30.503115 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.503849 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.505501 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.506188 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.506323 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:30.506406 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:30.506776 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 00:47:30.506878 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 00:47:30.507042 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:30.501795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53439 (local address 127.0.98.3:42585)
0927 00:47:30.501954 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:30.501958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:30.502157 (+ 199us) server_negotiation.cc:407] Connection header received
0927 00:47:30.502246 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:30.502249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:30.502316 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:30.502428 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:30.502774 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.503650 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.505502 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.506216 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.506323 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:30.506406 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:30.506776 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 00:47:30.506871 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:47:30.507133 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:30.501658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42585 (local address 127.0.98.2:53439)
0927 00:47:30.501956 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:30.501973 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:30.502128 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:30.502455 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:30.502459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:30.502481 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:30.502650 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:30.502661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.503754 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.503756 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:30.505314 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:30.505329 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.506397 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.506401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:30.506590 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:30.506650 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 00:47:30.506739 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 00:47:30.506902 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":169,"thread_start_us":119,"threads_started":1}
I20260927 00:47:30.507133 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:30.502118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40447 (local address 127.0.98.2:34367)
0927 00:47:30.502490 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:30.502502 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:30.502591 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:30.502920 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:30.502922 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:30.502938 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:30.503015 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:30.503020 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.503943 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.503946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:30.505314 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:30.505329 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.506406 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.506410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:30.506602 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:30.506646 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 00:47:30.506739 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 00:47:30.506902 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":128,"thread_start_us":88,"threads_started":1}
I20260927 00:47:30.508090 8074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda3caa885244130b0a7358f0d545a94" candidate_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09c44699d7f34db688c84b5d2e338142" is_pre_election: true
I20260927 00:47:30.508167 8354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda3caa885244130b0a7358f0d545a94" candidate_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" is_pre_election: true
I20260927 00:47:30.508612 8354 raft_consensus.cc:2454] T cda3caa885244130b0a7358f0d545a94 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0ae7faedd48405bbc5255ae0acbc6e2 in term 0.
I20260927 00:47:30.508612 8074 raft_consensus.cc:2454] T cda3caa885244130b0a7358f0d545a94 P 09c44699d7f34db688c84b5d2e338142 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0ae7faedd48405bbc5255ae0acbc6e2 in term 0.
I20260927 00:47:30.509271 8148 leader_election.cc:302] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f4a4619b3e04e6e8e99a20b640d4650, a0ae7faedd48405bbc5255ae0acbc6e2; no voters:
I20260927 00:47:30.509621 8436 raft_consensus.cc:2781] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:47:30.509717 8436 raft_consensus.cc:482] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:47:30.509760 8436 raft_consensus.cc:3037] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:30.511168 8436 raft_consensus.cc:504] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } } peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } }
I20260927 00:47:30.511641 8436 leader_election.cc:288] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2 [CANDIDATE]: Term 1 election: Requested vote from peers 2f4a4619b3e04e6e8e99a20b640d4650 (127.0.98.3:42585), 09c44699d7f34db688c84b5d2e338142 (127.0.98.1:40447)
I20260927 00:47:30.512205 8354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda3caa885244130b0a7358f0d545a94" candidate_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f4a4619b3e04e6e8e99a20b640d4650"
I20260927 00:47:30.512399 8074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda3caa885244130b0a7358f0d545a94" candidate_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09c44699d7f34db688c84b5d2e338142"
I20260927 00:47:30.512447 8354 raft_consensus.cc:3037] T cda3caa885244130b0a7358f0d545a94 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:30.512604 8074 raft_consensus.cc:3037] T cda3caa885244130b0a7358f0d545a94 P 09c44699d7f34db688c84b5d2e338142 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:30.514079 8074 raft_consensus.cc:2454] T cda3caa885244130b0a7358f0d545a94 P 09c44699d7f34db688c84b5d2e338142 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0ae7faedd48405bbc5255ae0acbc6e2 in term 1.
I20260927 00:47:30.514400 8354 raft_consensus.cc:2454] T cda3caa885244130b0a7358f0d545a94 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0ae7faedd48405bbc5255ae0acbc6e2 in term 1.
I20260927 00:47:30.514684 8148 leader_election.cc:302] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c44699d7f34db688c84b5d2e338142, a0ae7faedd48405bbc5255ae0acbc6e2; no voters:
I20260927 00:47:30.514922 8436 raft_consensus.cc:2781] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:30.515192 8436 raft_consensus.cc:686] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 1 LEADER]: Becoming Leader. State: Replica: a0ae7faedd48405bbc5255ae0acbc6e2, State: Running, Role: LEADER
I20260927 00:47:30.515760 8436 consensus_queue.cc:237] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } } peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } }
I20260927 00:47:30.520110 7917 catalog_manager.cc:5495] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0ae7faedd48405bbc5255ae0acbc6e2 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } health_report { overall_health: HEALTHY } } }
I20260927 00:47:30.574770 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:30.570773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48422 (local address 127.0.98.2:35833)
0927 00:47:30.570942 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:30.570948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:30.570979 (+ 31us) server_negotiation.cc:407] Connection header received
0927 00:47:30.571085 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:30.571087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:30.571162 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:30.571290 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:47:30.571567 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.572145 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.573079 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.573317 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.573435 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:30.573745 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:47:30.574340 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:47:30.574488 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:30.574526 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 00:47:30.574593 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 00:47:30.582576 8074 raft_consensus.cc:1261] T cda3caa885244130b0a7358f0d545a94 P 09c44699d7f34db688c84b5d2e338142 [term 1 FOLLOWER]: Refusing update from remote peer a0ae7faedd48405bbc5255ae0acbc6e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:47:30.582576 8354 raft_consensus.cc:1261] T cda3caa885244130b0a7358f0d545a94 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 1 FOLLOWER]: Refusing update from remote peer a0ae7faedd48405bbc5255ae0acbc6e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:47:30.583442 8436 consensus_queue.cc:1035] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:47:30.583707 8439 consensus_queue.cc:1035] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:47:30.591635 8442 mvcc.cc:205] Tried to move back new op lower bound from 7333765327177408512 to 7333765326917222400. Current Snapshot: MvccSnapshot[applied={T|T < 7333765327177408512}]
I20260927 00:47:30.593963 8443 mvcc.cc:205] Tried to move back new op lower bound from 7333765327177408512 to 7333765326917222400. Current Snapshot: MvccSnapshot[applied={T|T < 7333765327177408512 or (T in {7333765327177408512})}]
I20260927 00:47:30.603905 7917 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:47:30.604182 7917 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:47:30.607096 7917 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7dec1acba204282bf94e1473f15c1bb: Sending DeleteTablet for 3 replicas of tablet cda3caa885244130b0a7358f0d545a94
I20260927 00:47:30.608208 8334 tablet_service.cc:1514] Processing DeleteTablet for tablet cda3caa885244130b0a7358f0d545a94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42700
I20260927 00:47:30.608208 8054 tablet_service.cc:1514] Processing DeleteTablet for tablet cda3caa885244130b0a7358f0d545a94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34832
I20260927 00:47:30.608482 8194 tablet_service.cc:1514] Processing DeleteTablet for tablet cda3caa885244130b0a7358f0d545a94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48412
I20260927 00:47:30.608950 8452 tablet_replica.cc:330] T cda3caa885244130b0a7358f0d545a94 P 2f4a4619b3e04e6e8e99a20b640d4650: stopping tablet replica
I20260927 00:47:30.609243 8454 tablet_replica.cc:330] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2: stopping tablet replica
I20260927 00:47:30.609411 8452 raft_consensus.cc:2229] T cda3caa885244130b0a7358f0d545a94 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:47:30.609606 8454 raft_consensus.cc:2229] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:47:30.610075 8453 tablet_replica.cc:330] T cda3caa885244130b0a7358f0d545a94 P 09c44699d7f34db688c84b5d2e338142: stopping tablet replica
I20260927 00:47:30.609966 7917 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 00:47:30.610123 8454 raft_consensus.cc:2258] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 00:47:30.610512 7917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:47:30.610687 8453 raft_consensus.cc:2229] T cda3caa885244130b0a7358f0d545a94 P 09c44699d7f34db688c84b5d2e338142 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:47:30.611047 8453 raft_consensus.cc:2258] T cda3caa885244130b0a7358f0d545a94 P 09c44699d7f34db688c84b5d2e338142 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:30.611627 8452 raft_consensus.cc:2258] T cda3caa885244130b0a7358f0d545a94 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:30.613598 8452 ts_tablet_manager.cc:1905] T cda3caa885244130b0a7358f0d545a94 P 2f4a4619b3e04e6e8e99a20b640d4650: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:30.614271 8453 ts_tablet_manager.cc:1905] T cda3caa885244130b0a7358f0d545a94 P 09c44699d7f34db688c84b5d2e338142: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:30.613631 8454 ts_tablet_manager.cc:1905] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:30.617014 8452 ts_tablet_manager.cc:1918] T cda3caa885244130b0a7358f0d545a94 P 2f4a4619b3e04e6e8e99a20b640d4650: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:47:30.617136 8452 log.cc:1196] T cda3caa885244130b0a7358f0d545a94 P 2f4a4619b3e04e6e8e99a20b640d4650: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-2/wal/wals/cda3caa885244130b0a7358f0d545a94
I20260927 00:47:30.617475 8452 ts_tablet_manager.cc:1939] T cda3caa885244130b0a7358f0d545a94 P 2f4a4619b3e04e6e8e99a20b640d4650: Deleting consensus metadata
I20260927 00:47:30.618738 8054 tablet_service.cc:1467] Processing CreateTablet for tablet 2a28ed30afd4431dbc366c16dd2b0d30 (DEFAULT_TABLE table=test-table [id=a97130346e4f4acd91b56e2b34915f81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:30.619094 8054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a28ed30afd4431dbc366c16dd2b0d30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:30.620550 8453 ts_tablet_manager.cc:1918] T cda3caa885244130b0a7358f0d545a94 P 09c44699d7f34db688c84b5d2e338142: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:47:30.620676 8453 log.cc:1196] T cda3caa885244130b0a7358f0d545a94 P 09c44699d7f34db688c84b5d2e338142: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-0/wal/wals/cda3caa885244130b0a7358f0d545a94
I20260927 00:47:30.620879 8334 tablet_service.cc:1467] Processing CreateTablet for tablet 2a28ed30afd4431dbc366c16dd2b0d30 (DEFAULT_TABLE table=test-table [id=a97130346e4f4acd91b56e2b34915f81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:30.621032 8453 ts_tablet_manager.cc:1939] T cda3caa885244130b0a7358f0d545a94 P 09c44699d7f34db688c84b5d2e338142: Deleting consensus metadata
I20260927 00:47:30.621196 8334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a28ed30afd4431dbc366c16dd2b0d30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:30.621727 8454 ts_tablet_manager.cc:1918] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:47:30.621899 8454 log.cc:1196] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-1/wal/wals/cda3caa885244130b0a7358f0d545a94
I20260927 00:47:30.622287 8454 ts_tablet_manager.cc:1939] T cda3caa885244130b0a7358f0d545a94 P a0ae7faedd48405bbc5255ae0acbc6e2: Deleting consensus metadata
I20260927 00:47:30.623478 8430 tablet_bootstrap.cc:492] T 2a28ed30afd4431dbc366c16dd2b0d30 P 2f4a4619b3e04e6e8e99a20b640d4650: Bootstrap starting.
I20260927 00:47:30.623785 7904 catalog_manager.cc:4841] TS 09c44699d7f34db688c84b5d2e338142 (127.0.98.1:40447): tablet cda3caa885244130b0a7358f0d545a94 (table test-table [id=af6ff38eb9a745acbd8036ae883ee646]) successfully deleted
I20260927 00:47:30.624053 7904 catalog_manager.cc:4841] TS 2f4a4619b3e04e6e8e99a20b640d4650 (127.0.98.3:42585): tablet cda3caa885244130b0a7358f0d545a94 (table test-table [id=af6ff38eb9a745acbd8036ae883ee646]) successfully deleted
I20260927 00:47:30.624162 7903 catalog_manager.cc:4841] TS a0ae7faedd48405bbc5255ae0acbc6e2 (127.0.98.2:35833): tablet cda3caa885244130b0a7358f0d545a94 (table test-table [id=af6ff38eb9a745acbd8036ae883ee646]) successfully deleted
I20260927 00:47:30.624749 8430 tablet_bootstrap.cc:654] T 2a28ed30afd4431dbc366c16dd2b0d30 P 2f4a4619b3e04e6e8e99a20b640d4650: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:30.624850 8194 tablet_service.cc:1467] Processing CreateTablet for tablet 2a28ed30afd4431dbc366c16dd2b0d30 (DEFAULT_TABLE table=test-table [id=a97130346e4f4acd91b56e2b34915f81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:30.625144 8194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a28ed30afd4431dbc366c16dd2b0d30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:30.626116 8428 tablet_bootstrap.cc:492] T 2a28ed30afd4431dbc366c16dd2b0d30 P 09c44699d7f34db688c84b5d2e338142: Bootstrap starting.
I20260927 00:47:30.627291 8428 tablet_bootstrap.cc:654] T 2a28ed30afd4431dbc366c16dd2b0d30 P 09c44699d7f34db688c84b5d2e338142: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:30.627771 8429 tablet_bootstrap.cc:492] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2: Bootstrap starting.
I20260927 00:47:30.628059 8430 tablet_bootstrap.cc:492] T 2a28ed30afd4431dbc366c16dd2b0d30 P 2f4a4619b3e04e6e8e99a20b640d4650: No bootstrap required, opened a new log
I20260927 00:47:30.628146 8430 ts_tablet_manager.cc:1397] T 2a28ed30afd4431dbc366c16dd2b0d30 P 2f4a4619b3e04e6e8e99a20b640d4650: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260927 00:47:30.628718 8430 raft_consensus.cc:348] T 2a28ed30afd4431dbc366c16dd2b0d30 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:30.628822 8429 tablet_bootstrap.cc:654] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:30.628858 8430 raft_consensus.cc:374] T 2a28ed30afd4431dbc366c16dd2b0d30 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:30.628893 8430 raft_consensus.cc:729] T 2a28ed30afd4431dbc366c16dd2b0d30 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f4a4619b3e04e6e8e99a20b640d4650, State: Initialized, Role: FOLLOWER
I20260927 00:47:30.629035 8430 consensus_queue.cc:260] T 2a28ed30afd4431dbc366c16dd2b0d30 P 2f4a4619b3e04e6e8e99a20b640d4650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:30.629315 8430 ts_tablet_manager.cc:1428] T 2a28ed30afd4431dbc366c16dd2b0d30 P 2f4a4619b3e04e6e8e99a20b640d4650: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:47:30.630826 8429 tablet_bootstrap.cc:492] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2: No bootstrap required, opened a new log
I20260927 00:47:30.630921 8429 ts_tablet_manager.cc:1397] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 00:47:30.631429 8429 raft_consensus.cc:348] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:30.631560 8429 raft_consensus.cc:374] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:30.631600 8429 raft_consensus.cc:729] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0ae7faedd48405bbc5255ae0acbc6e2, State: Initialized, Role: FOLLOWER
I20260927 00:47:30.631747 8429 consensus_queue.cc:260] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:30.632023 8429 ts_tablet_manager.cc:1428] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:47:30.633133 8428 tablet_bootstrap.cc:492] T 2a28ed30afd4431dbc366c16dd2b0d30 P 09c44699d7f34db688c84b5d2e338142: No bootstrap required, opened a new log
I20260927 00:47:30.633260 8428 ts_tablet_manager.cc:1397] T 2a28ed30afd4431dbc366c16dd2b0d30 P 09c44699d7f34db688c84b5d2e338142: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260927 00:47:30.633998 8428 raft_consensus.cc:348] T 2a28ed30afd4431dbc366c16dd2b0d30 P 09c44699d7f34db688c84b5d2e338142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:30.634133 8428 raft_consensus.cc:374] T 2a28ed30afd4431dbc366c16dd2b0d30 P 09c44699d7f34db688c84b5d2e338142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:30.634166 8428 raft_consensus.cc:729] T 2a28ed30afd4431dbc366c16dd2b0d30 P 09c44699d7f34db688c84b5d2e338142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c44699d7f34db688c84b5d2e338142, State: Initialized, Role: FOLLOWER
I20260927 00:47:30.634295 8428 consensus_queue.cc:260] T 2a28ed30afd4431dbc366c16dd2b0d30 P 09c44699d7f34db688c84b5d2e338142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:30.634563 8428 ts_tablet_manager.cc:1428] T 2a28ed30afd4431dbc366c16dd2b0d30 P 09c44699d7f34db688c84b5d2e338142: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260927 00:47:30.810554 8447 raft_consensus.cc:482] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:47:30.810784 8447 raft_consensus.cc:504] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:30.811400 8447 leader_election.cc:288] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09c44699d7f34db688c84b5d2e338142 (127.0.98.1:40447), 2f4a4619b3e04e6e8e99a20b640d4650 (127.0.98.3:42585)
I20260927 00:47:30.812111 8354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a28ed30afd4431dbc366c16dd2b0d30" candidate_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" is_pre_election: true
I20260927 00:47:30.812119 8074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a28ed30afd4431dbc366c16dd2b0d30" candidate_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09c44699d7f34db688c84b5d2e338142" is_pre_election: true
I20260927 00:47:30.812404 8354 raft_consensus.cc:2454] T 2a28ed30afd4431dbc366c16dd2b0d30 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0ae7faedd48405bbc5255ae0acbc6e2 in term 0.
I20260927 00:47:30.812418 8074 raft_consensus.cc:2454] T 2a28ed30afd4431dbc366c16dd2b0d30 P 09c44699d7f34db688c84b5d2e338142 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0ae7faedd48405bbc5255ae0acbc6e2 in term 0.
I20260927 00:47:30.812852 8148 leader_election.cc:302] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f4a4619b3e04e6e8e99a20b640d4650, a0ae7faedd48405bbc5255ae0acbc6e2; no voters:
I20260927 00:47:30.813076 8447 raft_consensus.cc:2781] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:47:30.813140 8447 raft_consensus.cc:482] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:47:30.813180 8447 raft_consensus.cc:3037] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:30.814693 8447 raft_consensus.cc:504] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:30.815171 8447 leader_election.cc:288] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2 [CANDIDATE]: Term 1 election: Requested vote from peers 09c44699d7f34db688c84b5d2e338142 (127.0.98.1:40447), 2f4a4619b3e04e6e8e99a20b640d4650 (127.0.98.3:42585)
I20260927 00:47:30.815721 8074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a28ed30afd4431dbc366c16dd2b0d30" candidate_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09c44699d7f34db688c84b5d2e338142"
I20260927 00:47:30.815721 8354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a28ed30afd4431dbc366c16dd2b0d30" candidate_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f4a4619b3e04e6e8e99a20b640d4650"
I20260927 00:47:30.815881 8354 raft_consensus.cc:3037] T 2a28ed30afd4431dbc366c16dd2b0d30 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:30.815881 8074 raft_consensus.cc:3037] T 2a28ed30afd4431dbc366c16dd2b0d30 P 09c44699d7f34db688c84b5d2e338142 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:30.817309 8354 raft_consensus.cc:2454] T 2a28ed30afd4431dbc366c16dd2b0d30 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0ae7faedd48405bbc5255ae0acbc6e2 in term 1.
I20260927 00:47:30.817309 8074 raft_consensus.cc:2454] T 2a28ed30afd4431dbc366c16dd2b0d30 P 09c44699d7f34db688c84b5d2e338142 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0ae7faedd48405bbc5255ae0acbc6e2 in term 1.
I20260927 00:47:30.817703 8148 leader_election.cc:302] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f4a4619b3e04e6e8e99a20b640d4650, a0ae7faedd48405bbc5255ae0acbc6e2; no voters:
I20260927 00:47:30.817900 8447 raft_consensus.cc:2781] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:30.817992 8447 raft_consensus.cc:686] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 1 LEADER]: Becoming Leader. State: Replica: a0ae7faedd48405bbc5255ae0acbc6e2, State: Running, Role: LEADER
I20260927 00:47:30.818183 8447 consensus_queue.cc:237] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:30.820509 7914 catalog_manager.cc:5495] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0ae7faedd48405bbc5255ae0acbc6e2 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
Sep 27 00:47:30 dist-test-slave-86rs krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470049, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:47:30.868420 8464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:30.858708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38268 (local address 127.0.98.62:46561)
0927 00:47:30.859029 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:30.859035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:30.859165 (+ 130us) server_negotiation.cc:407] Connection header received
0927 00:47:30.859356 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:30.859360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:30.859409 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:30.859529 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:30.860403 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.860993 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.861852 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.862093 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.862210 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:30.864839 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:30.864842 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:30.864845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:30.864865 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:30.866340 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:30.866881 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:30.866887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:30.866889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:30.866982 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:30.867274 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:30.867279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:30.867281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:30.867501 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:30.867609 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:30.867943 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 00:47:30.868131 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":223,"thread_start_us":129,"threads_started":1}
I20260927 00:47:30.868420 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:30.858667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46561 (local address 127.0.0.1:38268)
0927 00:47:30.859049 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:30.859066 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:30.859245 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:30.859550 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:30.859552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:30.860109 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:30.860221 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:30.860278 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.861118 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.861122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:30.861730 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:30.861736 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:30.862216 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:30.862219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:30.862292 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:30.863015 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:30.863027 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:30.864676 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:30.866484 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:30.866496 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:30.866506 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:30.866774 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:30.867082 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:30.867086 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:30.867087 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:30.867180 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:30.867629 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:30.867637 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:30.867764 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 00:47:30.867885 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 00:47:30.868188 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":128,"thread_start_us":120,"threads_started":1}
I20260927 00:47:30.873291 7914 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38268:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 00:47:30.874176 7914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:47:30.880061 8334 tablet_service.cc:1467] Processing CreateTablet for tablet 6e606461469e4306b2c9ec6e28354b78 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d88ede4e1f554558826bdbfad69498f7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:47:30.880007 8194 tablet_service.cc:1467] Processing CreateTablet for tablet 6e606461469e4306b2c9ec6e28354b78 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d88ede4e1f554558826bdbfad69498f7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:47:30.880009 8054 tablet_service.cc:1467] Processing CreateTablet for tablet 6e606461469e4306b2c9ec6e28354b78 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d88ede4e1f554558826bdbfad69498f7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:47:30.880585 8194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e606461469e4306b2c9ec6e28354b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:30.880586 8054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e606461469e4306b2c9ec6e28354b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:30.880585 8334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e606461469e4306b2c9ec6e28354b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:30.883844 8428 tablet_bootstrap.cc:492] T 6e606461469e4306b2c9ec6e28354b78 P 09c44699d7f34db688c84b5d2e338142: Bootstrap starting.
I20260927 00:47:30.883901 8430 tablet_bootstrap.cc:492] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650: Bootstrap starting.
I20260927 00:47:30.884557 8429 tablet_bootstrap.cc:492] T 6e606461469e4306b2c9ec6e28354b78 P a0ae7faedd48405bbc5255ae0acbc6e2: Bootstrap starting.
I20260927 00:47:30.885021 8430 tablet_bootstrap.cc:654] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:30.885020 8428 tablet_bootstrap.cc:654] T 6e606461469e4306b2c9ec6e28354b78 P 09c44699d7f34db688c84b5d2e338142: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:30.885350 8429 tablet_bootstrap.cc:654] T 6e606461469e4306b2c9ec6e28354b78 P a0ae7faedd48405bbc5255ae0acbc6e2: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:30.886574 8428 tablet_bootstrap.cc:492] T 6e606461469e4306b2c9ec6e28354b78 P 09c44699d7f34db688c84b5d2e338142: No bootstrap required, opened a new log
I20260927 00:47:30.886688 8428 ts_tablet_manager.cc:1397] T 6e606461469e4306b2c9ec6e28354b78 P 09c44699d7f34db688c84b5d2e338142: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:47:30.886760 8429 tablet_bootstrap.cc:492] T 6e606461469e4306b2c9ec6e28354b78 P a0ae7faedd48405bbc5255ae0acbc6e2: No bootstrap required, opened a new log
I20260927 00:47:30.886842 8430 tablet_bootstrap.cc:492] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650: No bootstrap required, opened a new log
I20260927 00:47:30.886853 8429 ts_tablet_manager.cc:1397] T 6e606461469e4306b2c9ec6e28354b78 P a0ae7faedd48405bbc5255ae0acbc6e2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 00:47:30.886935 8430 ts_tablet_manager.cc:1397] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:47:30.887251 8428 raft_consensus.cc:348] T 6e606461469e4306b2c9ec6e28354b78 P 09c44699d7f34db688c84b5d2e338142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:30.887292 8429 raft_consensus.cc:348] T 6e606461469e4306b2c9ec6e28354b78 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:30.887398 8428 raft_consensus.cc:374] T 6e606461469e4306b2c9ec6e28354b78 P 09c44699d7f34db688c84b5d2e338142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:30.887415 8429 raft_consensus.cc:374] T 6e606461469e4306b2c9ec6e28354b78 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:30.887439 8428 raft_consensus.cc:729] T 6e606461469e4306b2c9ec6e28354b78 P 09c44699d7f34db688c84b5d2e338142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c44699d7f34db688c84b5d2e338142, State: Initialized, Role: FOLLOWER
I20260927 00:47:30.887461 8429 raft_consensus.cc:729] T 6e606461469e4306b2c9ec6e28354b78 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0ae7faedd48405bbc5255ae0acbc6e2, State: Initialized, Role: FOLLOWER
I20260927 00:47:30.887575 8428 consensus_queue.cc:260] T 6e606461469e4306b2c9ec6e28354b78 P 09c44699d7f34db688c84b5d2e338142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:30.887588 8429 consensus_queue.cc:260] T 6e606461469e4306b2c9ec6e28354b78 P a0ae7faedd48405bbc5255ae0acbc6e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:30.887401 8430 raft_consensus.cc:348] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:30.887708 8430 raft_consensus.cc:374] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:30.887749 8430 raft_consensus.cc:729] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f4a4619b3e04e6e8e99a20b640d4650, State: Initialized, Role: FOLLOWER
I20260927 00:47:30.887902 8429 ts_tablet_manager.cc:1428] T 6e606461469e4306b2c9ec6e28354b78 P a0ae7faedd48405bbc5255ae0acbc6e2: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260927 00:47:30.887920 8428 ts_tablet_manager.cc:1428] T 6e606461469e4306b2c9ec6e28354b78 P 09c44699d7f34db688c84b5d2e338142: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 00:47:30.887877 8430 consensus_queue.cc:260] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:30.888177 8434 tablet_replica.cc:439] T 6e606461469e4306b2c9ec6e28354b78 P 09c44699d7f34db688c84b5d2e338142: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } } }
I20260927 00:47:30.888199 8447 tablet_replica.cc:439] T 6e606461469e4306b2c9ec6e28354b78 P a0ae7faedd48405bbc5255ae0acbc6e2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } } }
I20260927 00:47:30.888370 8447 tablet_replica.cc:442] T 6e606461469e4306b2c9ec6e28354b78 P a0ae7faedd48405bbc5255ae0acbc6e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:47:30.888370 8434 tablet_replica.cc:442] T 6e606461469e4306b2c9ec6e28354b78 P 09c44699d7f34db688c84b5d2e338142: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:47:30.888135 8435 tablet_replica.cc:439] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } } }
I20260927 00:47:30.888876 8435 tablet_replica.cc:442] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:47:30.889189 8430 ts_tablet_manager.cc:1428] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 00:47:31.065395 8435 raft_consensus.cc:482] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:47:31.065622 8435 raft_consensus.cc:504] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:31.067176 8435 leader_election.cc:288] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09c44699d7f34db688c84b5d2e338142 (127.0.98.1:40447), a0ae7faedd48405bbc5255ae0acbc6e2 (127.0.98.2:35833)
I20260927 00:47:31.072425 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:31.067464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40447 (local address 127.0.98.3:38843)
0927 00:47:31.067804 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:31.067828 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:31.067978 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:31.068444 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:31.068448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:31.068478 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:31.068685 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:31.068698 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.069680 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.069686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:31.071059 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:31.071070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.071861 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.071865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:31.071978 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:31.072010 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 00:47:31.072081 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 00:47:31.072196 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":6,"thread_start_us":106,"threads_started":1}
I20260927 00:47:31.072543 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:31.067904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45157 (local address 127.0.98.2:35833)
0927 00:47:31.068055 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:31.068059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:31.068329 (+ 270us) server_negotiation.cc:407] Connection header received
0927 00:47:31.068490 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:31.068493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:31.068543 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:31.068585 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:31.069022 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.069897 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.071219 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.071900 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.072022 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:31.072121 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:31.072299 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 00:47:31.072387 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:47:31.072851 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:31.067630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38843 (local address 127.0.98.1:40447)
0927 00:47:31.067977 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:31.067983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:31.068005 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:47:31.068097 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:31.068101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:31.068178 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:31.068319 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:31.068854 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.069536 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.071223 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.071730 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.071859 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:31.071948 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:31.072540 (+ 592us) server_negotiation.cc:299] Negotiation successful
0927 00:47:31.072662 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":103,"threads_started":1}
I20260927 00:47:31.073180 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:31.067616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35833 (local address 127.0.98.3:45157)
0927 00:47:31.068179 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:31.068194 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:31.068335 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:31.068718 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:31.068720 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:31.068733 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:31.068880 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:31.068887 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.070003 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.070006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:31.071069 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:31.071075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.072017 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.072021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:31.072123 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:31.072142 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 00:47:31.072901 (+ 759us) client_negotiation.cc:240] Negotiation successful
0927 00:47:31.073024 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":121,"threads_started":1}
I20260927 00:47:31.073402 8074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e606461469e4306b2c9ec6e28354b78" candidate_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09c44699d7f34db688c84b5d2e338142" is_pre_election: true
I20260927 00:47:31.073635 8074 raft_consensus.cc:2454] T 6e606461469e4306b2c9ec6e28354b78 P 09c44699d7f34db688c84b5d2e338142 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f4a4619b3e04e6e8e99a20b640d4650 in term 0.
I20260927 00:47:31.074095 8288 leader_election.cc:302] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c44699d7f34db688c84b5d2e338142, 2f4a4619b3e04e6e8e99a20b640d4650; no voters:
I20260927 00:47:31.074084 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e606461469e4306b2c9ec6e28354b78" candidate_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" is_pre_election: true
I20260927 00:47:31.074424 8435 raft_consensus.cc:2781] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:47:31.074415 8214 raft_consensus.cc:2454] T 6e606461469e4306b2c9ec6e28354b78 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f4a4619b3e04e6e8e99a20b640d4650 in term 0.
I20260927 00:47:31.074504 8435 raft_consensus.cc:482] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:47:31.074548 8435 raft_consensus.cc:3037] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:31.075946 8435 raft_consensus.cc:504] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:31.076422 8435 leader_election.cc:288] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650 [CANDIDATE]: Term 1 election: Requested vote from peers 09c44699d7f34db688c84b5d2e338142 (127.0.98.1:40447), a0ae7faedd48405bbc5255ae0acbc6e2 (127.0.98.2:35833)
I20260927 00:47:31.076823 8074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e606461469e4306b2c9ec6e28354b78" candidate_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09c44699d7f34db688c84b5d2e338142"
I20260927 00:47:31.076983 8074 raft_consensus.cc:3037] T 6e606461469e4306b2c9ec6e28354b78 P 09c44699d7f34db688c84b5d2e338142 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:31.078513 8074 raft_consensus.cc:2454] T 6e606461469e4306b2c9ec6e28354b78 P 09c44699d7f34db688c84b5d2e338142 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f4a4619b3e04e6e8e99a20b640d4650 in term 1.
I20260927 00:47:31.078862 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e606461469e4306b2c9ec6e28354b78" candidate_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2"
I20260927 00:47:31.078910 8288 leader_election.cc:302] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c44699d7f34db688c84b5d2e338142, 2f4a4619b3e04e6e8e99a20b640d4650; no voters:
I20260927 00:47:31.079048 8214 raft_consensus.cc:3037] T 6e606461469e4306b2c9ec6e28354b78 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:31.079125 8435 raft_consensus.cc:2781] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:31.079365 8435 raft_consensus.cc:686] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 1 LEADER]: Becoming Leader. State: Replica: 2f4a4619b3e04e6e8e99a20b640d4650, State: Running, Role: LEADER
I20260927 00:47:31.079747 8435 consensus_queue.cc:237] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:31.080535 8214 raft_consensus.cc:2454] T 6e606461469e4306b2c9ec6e28354b78 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f4a4619b3e04e6e8e99a20b640d4650 in term 1.
I20260927 00:47:31.081439 8472 tablet_replica.cc:439] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650: TxnStatusTablet state changed. Reason: New leader 2f4a4619b3e04e6e8e99a20b640d4650. Latest consensus state: current_term: 1 leader_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } } }
I20260927 00:47:31.081575 8472 tablet_replica.cc:442] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:47:31.081928 8474 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 00:47:31.082132 8474 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 00:47:31.082995 7916 catalog_manager.cc:5495] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f4a4619b3e04e6e8e99a20b640d4650 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20260927 00:47:31.137428 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:31.131835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42585 (local address 127.0.0.1:42702)
0927 00:47:31.132061 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:31.132081 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:31.132277 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:31.132924 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:31.132927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:31.132941 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:47:31.133085 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:31.133092 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.133976 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.133980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:31.135033 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:31.135046 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.135534 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.135538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:31.135646 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:31.135723 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:47:31.137018 (+ 1295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:47:31.137043 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:47:31.137128 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 00:47:31.137258 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 00:47:31.137458 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:31.131992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42702 (local address 127.0.98.3:42585)
0927 00:47:31.132525 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:31.132530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:31.132551 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:47:31.132638 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:31.132641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:31.132741 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:31.132878 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:47:31.133210 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.133864 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.135176 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.135417 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.135500 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:31.135883 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:47:31.136586 (+ 703us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:47:31.137042 (+ 456us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:31.137157 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 00:47:31.137243 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":113,"threads_started":1}
I20260927 00:47:31.141871 8074 raft_consensus.cc:1261] T 6e606461469e4306b2c9ec6e28354b78 P 09c44699d7f34db688c84b5d2e338142 [term 1 FOLLOWER]: Refusing update from remote peer 2f4a4619b3e04e6e8e99a20b640d4650: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:47:31.142087 8214 raft_consensus.cc:1261] T 6e606461469e4306b2c9ec6e28354b78 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 1 FOLLOWER]: Refusing update from remote peer 2f4a4619b3e04e6e8e99a20b640d4650: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:47:31.142599 8472 consensus_queue.cc:1035] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:47:31.142822 8435 consensus_queue.cc:1035] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:47:31.145081 8447 tablet_replica.cc:439] T 6e606461469e4306b2c9ec6e28354b78 P a0ae7faedd48405bbc5255ae0acbc6e2: TxnStatusTablet state changed. Reason: New leader 2f4a4619b3e04e6e8e99a20b640d4650. Latest consensus state: current_term: 1 leader_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } } }
I20260927 00:47:31.145232 8447 tablet_replica.cc:442] T 6e606461469e4306b2c9ec6e28354b78 P a0ae7faedd48405bbc5255ae0acbc6e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:47:31.145411 8434 tablet_replica.cc:439] T 6e606461469e4306b2c9ec6e28354b78 P 09c44699d7f34db688c84b5d2e338142: TxnStatusTablet state changed. Reason: New leader 2f4a4619b3e04e6e8e99a20b640d4650. Latest consensus state: current_term: 1 leader_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } } }
I20260927 00:47:31.145545 8434 tablet_replica.cc:442] T 6e606461469e4306b2c9ec6e28354b78 P 09c44699d7f34db688c84b5d2e338142: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:47:31.148130 8435 tablet_replica.cc:439] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } } }
I20260927 00:47:31.148319 8435 tablet_replica.cc:442] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:47:31.149263 8434 tablet_replica.cc:439] T 6e606461469e4306b2c9ec6e28354b78 P 09c44699d7f34db688c84b5d2e338142: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } } }
I20260927 00:47:31.149422 8434 tablet_replica.cc:442] T 6e606461469e4306b2c9ec6e28354b78 P 09c44699d7f34db688c84b5d2e338142: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:47:31.148175 8472 tablet_replica.cc:439] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } } }
I20260927 00:47:31.150677 8472 tablet_replica.cc:442] T 6e606461469e4306b2c9ec6e28354b78 P 2f4a4619b3e04e6e8e99a20b640d4650: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:47:31.151777 8447 tablet_replica.cc:439] T 6e606461469e4306b2c9ec6e28354b78 P a0ae7faedd48405bbc5255ae0acbc6e2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } } }
I20260927 00:47:31.151940 8447 tablet_replica.cc:442] T 6e606461469e4306b2c9ec6e28354b78 P a0ae7faedd48405bbc5255ae0acbc6e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:47:31.152420 8479 mvcc.cc:205] Tried to move back new op lower bound from 7333765329470074880 to 7333765329226493952. Current Snapshot: MvccSnapshot[applied={T|T < 7333765329470074880}]
I20260927 00:47:31.167474 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:31.163043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53299 (local address 127.0.98.3:42585)
0927 00:47:31.163245 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:31.163250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:31.163517 (+ 267us) server_negotiation.cc:407] Connection header received
0927 00:47:31.163689 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:31.163691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:31.163757 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:31.163831 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:47:31.164172 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.164800 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.166119 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.166349 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.166427 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:31.166854 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:47:31.166984 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:47:31.167101 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:31.167248 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 00:47:31.167325 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:47:31.167511 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:31.162921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42585 (local address 127.0.98.2:53299)
0927 00:47:31.163410 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:31.163427 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:31.163573 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:31.163878 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:31.163882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:31.163892 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:47:31.164030 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:31.164037 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.164920 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.164923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:31.165983 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:31.165995 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.166466 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.166470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:31.166570 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:31.166655 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:47:31.167122 (+ 467us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:47:31.167129 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:47:31.167203 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 00:47:31.167321 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":156,"threads_started":1}
I20260927 00:47:31.176960 8354 raft_consensus.cc:1261] T 2a28ed30afd4431dbc366c16dd2b0d30 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 1 FOLLOWER]: Refusing update from remote peer a0ae7faedd48405bbc5255ae0acbc6e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:47:31.176959 8074 raft_consensus.cc:1261] T 2a28ed30afd4431dbc366c16dd2b0d30 P 09c44699d7f34db688c84b5d2e338142 [term 1 FOLLOWER]: Refusing update from remote peer a0ae7faedd48405bbc5255ae0acbc6e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:47:31.177512 8439 consensus_queue.cc:1035] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:47:31.177762 8447 consensus_queue.cc:1035] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:47:31.206375 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:31.201338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45247 (local address 127.0.98.2:35833)
0927 00:47:31.201477 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:31.201484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:31.201781 (+ 297us) server_negotiation.cc:407] Connection header received
0927 00:47:31.201991 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:31.201994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:31.202060 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:31.202113 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:47:31.202543 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.203381 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.204843 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.205201 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.205283 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:31.205730 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:47:31.205842 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:47:31.205941 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:31.206110 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 00:47:31.206172 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:47:31.206383 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:31.201236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35833 (local address 127.0.98.3:45247)
0927 00:47:31.201688 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:31.201705 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:31.201866 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:31.202266 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:31.202269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:31.202278 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:47:31.202408 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:31.202414 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.203561 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.203564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:31.204700 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:31.204714 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.205426 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.205429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:31.205531 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:31.205594 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:47:31.205960 (+ 366us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:47:31.205968 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 00:47:31.206045 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 00:47:31.206147 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":126,"threads_started":1}
I20260927 00:47:31.233299 7916 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:47:31.233561 7916 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:47:31.236352 7916 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7dec1acba204282bf94e1473f15c1bb: Sending DeleteTablet for 3 replicas of tablet 2a28ed30afd4431dbc366c16dd2b0d30
I20260927 00:47:31.237238 8194 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a28ed30afd4431dbc366c16dd2b0d30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48412
I20260927 00:47:31.237246 8054 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a28ed30afd4431dbc366c16dd2b0d30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34832
I20260927 00:47:31.237919 8334 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a28ed30afd4431dbc366c16dd2b0d30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42700
I20260927 00:47:31.238430 8499 tablet_replica.cc:330] T 2a28ed30afd4431dbc366c16dd2b0d30 P 2f4a4619b3e04e6e8e99a20b640d4650: stopping tablet replica
I20260927 00:47:31.238435 8498 tablet_replica.cc:330] T 2a28ed30afd4431dbc366c16dd2b0d30 P 09c44699d7f34db688c84b5d2e338142: stopping tablet replica
I20260927 00:47:31.238618 8498 raft_consensus.cc:2229] T 2a28ed30afd4431dbc366c16dd2b0d30 P 09c44699d7f34db688c84b5d2e338142 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:47:31.238685 8499 raft_consensus.cc:2229] T 2a28ed30afd4431dbc366c16dd2b0d30 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:47:31.238895 8498 raft_consensus.cc:2258] T 2a28ed30afd4431dbc366c16dd2b0d30 P 09c44699d7f34db688c84b5d2e338142 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:31.238915 8499 raft_consensus.cc:2258] T 2a28ed30afd4431dbc366c16dd2b0d30 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:31.239897 8498 ts_tablet_manager.cc:1905] T 2a28ed30afd4431dbc366c16dd2b0d30 P 09c44699d7f34db688c84b5d2e338142: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:31.239907 8499 ts_tablet_manager.cc:1905] T 2a28ed30afd4431dbc366c16dd2b0d30 P 2f4a4619b3e04e6e8e99a20b640d4650: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:31.243826 8498 ts_tablet_manager.cc:1918] T 2a28ed30afd4431dbc366c16dd2b0d30 P 09c44699d7f34db688c84b5d2e338142: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:47:31.243938 8499 ts_tablet_manager.cc:1918] T 2a28ed30afd4431dbc366c16dd2b0d30 P 2f4a4619b3e04e6e8e99a20b640d4650: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:47:31.244066 8498 log.cc:1196] T 2a28ed30afd4431dbc366c16dd2b0d30 P 09c44699d7f34db688c84b5d2e338142: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-0/wal/wals/2a28ed30afd4431dbc366c16dd2b0d30
I20260927 00:47:31.244107 8499 log.cc:1196] T 2a28ed30afd4431dbc366c16dd2b0d30 P 2f4a4619b3e04e6e8e99a20b640d4650: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-2/wal/wals/2a28ed30afd4431dbc366c16dd2b0d30
I20260927 00:47:31.244480 8499 ts_tablet_manager.cc:1939] T 2a28ed30afd4431dbc366c16dd2b0d30 P 2f4a4619b3e04e6e8e99a20b640d4650: Deleting consensus metadata
I20260927 00:47:31.244544 8498 ts_tablet_manager.cc:1939] T 2a28ed30afd4431dbc366c16dd2b0d30 P 09c44699d7f34db688c84b5d2e338142: Deleting consensus metadata
I20260927 00:47:31.245608 7904 catalog_manager.cc:4841] TS 2f4a4619b3e04e6e8e99a20b640d4650 (127.0.98.3:42585): tablet 2a28ed30afd4431dbc366c16dd2b0d30 (table test-table [id=a97130346e4f4acd91b56e2b34915f81]) successfully deleted
I20260927 00:47:31.245832 7904 catalog_manager.cc:4841] TS 09c44699d7f34db688c84b5d2e338142 (127.0.98.1:40447): tablet 2a28ed30afd4431dbc366c16dd2b0d30 (table test-table [id=a97130346e4f4acd91b56e2b34915f81]) successfully deleted
I20260927 00:47:31.247424 8497 tablet_replica.cc:330] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2: stopping tablet replica
I20260927 00:47:31.247638 8497 raft_consensus.cc:2229] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:47:31.247927 8497 raft_consensus.cc:2258] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:31.248875 8497 ts_tablet_manager.cc:1905] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:31.252777 8497 ts_tablet_manager.cc:1918] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:47:31.252913 8497 log.cc:1196] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-1/wal/wals/2a28ed30afd4431dbc366c16dd2b0d30
I20260927 00:47:31.253248 8497 ts_tablet_manager.cc:1939] T 2a28ed30afd4431dbc366c16dd2b0d30 P a0ae7faedd48405bbc5255ae0acbc6e2: Deleting consensus metadata
I20260927 00:47:31.254292 7903 catalog_manager.cc:4841] TS a0ae7faedd48405bbc5255ae0acbc6e2 (127.0.98.2:35833): tablet 2a28ed30afd4431dbc366c16dd2b0d30 (table test-table [id=a97130346e4f4acd91b56e2b34915f81]) successfully deleted
Sep 27 00:47:31 dist-test-slave-86rs krb5kdc[7872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:47:31 dist-test-slave-86rs krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:31 dist-test-slave-86rs krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 00:47:31.271513 8464 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:31.264162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33688 (local address 127.0.98.62:46561)
0927 00:47:31.264394 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:31.264401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:31.264679 (+ 278us) server_negotiation.cc:407] Connection header received
0927 00:47:31.264900 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:31.264904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:31.264979 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:31.265077 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:31.265911 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.266520 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.267461 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.267726 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.267838 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:31.271354 (+ 3516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33688: BlockingRecv error: recv got EOF from 127.0.0.1:33688 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
Sep 27 00:47:31 dist-test-slave-86rs krb5kdc[7872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:47:31.284794 8464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:31.273567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33704 (local address 127.0.98.62:46561)
0927 00:47:31.273727 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:31.273731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:31.273944 (+ 213us) server_negotiation.cc:407] Connection header received
0927 00:47:31.274133 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:31.274136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:31.274201 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:31.274291 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:31.274947 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.275519 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.276476 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.276741 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.276890 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:31.280855 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:31.280859 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:31.280861 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:31.280897 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:31.282681 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:31.283383 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:31.283386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:31.283387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:31.283468 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:31.283796 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:31.283798 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:31.283800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:31.284002 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:31.284119 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:31.284478 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 00:47:31.284608 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 00:47:31.288362 7916 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:47:31.288954 7916 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:47:31.294934 8054 tablet_service.cc:1467] Processing CreateTablet for tablet b7995f4663704f9f92147fd283f8a510 (DEFAULT_TABLE table=test-table [id=b234d69cb4ca479aae1234dfdc3b55f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:31.295073 8334 tablet_service.cc:1467] Processing CreateTablet for tablet b7995f4663704f9f92147fd283f8a510 (DEFAULT_TABLE table=test-table [id=b234d69cb4ca479aae1234dfdc3b55f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:31.295125 8194 tablet_service.cc:1467] Processing CreateTablet for tablet b7995f4663704f9f92147fd283f8a510 (DEFAULT_TABLE table=test-table [id=b234d69cb4ca479aae1234dfdc3b55f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:31.295411 8054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7995f4663704f9f92147fd283f8a510. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:31.295450 8194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7995f4663704f9f92147fd283f8a510. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:31.295450 8334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7995f4663704f9f92147fd283f8a510. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:31.298810 8430 tablet_bootstrap.cc:492] T b7995f4663704f9f92147fd283f8a510 P 2f4a4619b3e04e6e8e99a20b640d4650: Bootstrap starting.
I20260927 00:47:31.298813 8429 tablet_bootstrap.cc:492] T b7995f4663704f9f92147fd283f8a510 P a0ae7faedd48405bbc5255ae0acbc6e2: Bootstrap starting.
I20260927 00:47:31.299286 8428 tablet_bootstrap.cc:492] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142: Bootstrap starting.
I20260927 00:47:31.299929 8430 tablet_bootstrap.cc:654] T b7995f4663704f9f92147fd283f8a510 P 2f4a4619b3e04e6e8e99a20b640d4650: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:31.299961 8429 tablet_bootstrap.cc:654] T b7995f4663704f9f92147fd283f8a510 P a0ae7faedd48405bbc5255ae0acbc6e2: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:31.300778 8428 tablet_bootstrap.cc:654] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:31.301304 8430 tablet_bootstrap.cc:492] T b7995f4663704f9f92147fd283f8a510 P 2f4a4619b3e04e6e8e99a20b640d4650: No bootstrap required, opened a new log
I20260927 00:47:31.301410 8430 ts_tablet_manager.cc:1397] T b7995f4663704f9f92147fd283f8a510 P 2f4a4619b3e04e6e8e99a20b640d4650: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:47:31.301950 8430 raft_consensus.cc:348] T b7995f4663704f9f92147fd283f8a510 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:31.302135 8430 raft_consensus.cc:374] T b7995f4663704f9f92147fd283f8a510 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:31.302183 8430 raft_consensus.cc:729] T b7995f4663704f9f92147fd283f8a510 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f4a4619b3e04e6e8e99a20b640d4650, State: Initialized, Role: FOLLOWER
I20260927 00:47:31.302237 8428 tablet_bootstrap.cc:492] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142: No bootstrap required, opened a new log
I20260927 00:47:31.302343 8428 ts_tablet_manager.cc:1397] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:47:31.302316 8430 consensus_queue.cc:260] T b7995f4663704f9f92147fd283f8a510 P 2f4a4619b3e04e6e8e99a20b640d4650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:31.302538 8429 tablet_bootstrap.cc:492] T b7995f4663704f9f92147fd283f8a510 P a0ae7faedd48405bbc5255ae0acbc6e2: No bootstrap required, opened a new log
I20260927 00:47:31.302603 8430 ts_tablet_manager.cc:1428] T b7995f4663704f9f92147fd283f8a510 P 2f4a4619b3e04e6e8e99a20b640d4650: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:47:31.302639 8429 ts_tablet_manager.cc:1397] T b7995f4663704f9f92147fd283f8a510 P a0ae7faedd48405bbc5255ae0acbc6e2: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260927 00:47:31.302965 8428 raft_consensus.cc:348] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:31.303114 8428 raft_consensus.cc:374] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:31.303159 8428 raft_consensus.cc:729] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c44699d7f34db688c84b5d2e338142, State: Initialized, Role: FOLLOWER
I20260927 00:47:31.303150 8429 raft_consensus.cc:348] T b7995f4663704f9f92147fd283f8a510 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:31.303292 8429 raft_consensus.cc:374] T b7995f4663704f9f92147fd283f8a510 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:31.303332 8429 raft_consensus.cc:729] T b7995f4663704f9f92147fd283f8a510 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0ae7faedd48405bbc5255ae0acbc6e2, State: Initialized, Role: FOLLOWER
I20260927 00:47:31.303326 8428 consensus_queue.cc:260] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:31.303527 8429 consensus_queue.cc:260] T b7995f4663704f9f92147fd283f8a510 P a0ae7faedd48405bbc5255ae0acbc6e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:31.303692 8428 ts_tablet_manager.cc:1428] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:47:31.304221 8429 ts_tablet_manager.cc:1428] T b7995f4663704f9f92147fd283f8a510 P a0ae7faedd48405bbc5255ae0acbc6e2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 00:47:31.546108 8434 raft_consensus.cc:482] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:47:31.546295 8434 raft_consensus.cc:504] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:31.548153 8434 leader_election.cc:288] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0ae7faedd48405bbc5255ae0acbc6e2 (127.0.98.2:35833), 2f4a4619b3e04e6e8e99a20b640d4650 (127.0.98.3:42585)
I20260927 00:47:31.552138 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:31.548190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54647 (local address 127.0.98.2:35833)
0927 00:47:31.548392 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:31.548397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:31.548576 (+ 179us) server_negotiation.cc:407] Connection header received
0927 00:47:31.548862 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:31.548868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:31.548950 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:31.549050 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:31.549381 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.549957 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.551090 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.551525 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.551654 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:31.551737 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:31.551906 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 00:47:31.551989 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 00:47:31.552166 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:31.548038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35833 (local address 127.0.98.1:54647)
0927 00:47:31.548458 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:31.548475 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:31.548687 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:31.549063 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:31.549066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:31.549084 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:31.549264 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:31.549270 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.550074 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.550080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:31.550951 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:31.550968 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.551654 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.551657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:31.551762 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:31.551796 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 00:47:31.551890 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 00:47:31.551981 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":3,"thread_start_us":111,"threads_started":1}
I20260927 00:47:31.552230 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:31.548411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42585 (local address 127.0.98.1:51025)
0927 00:47:31.548776 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:31.548791 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:31.548907 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:31.549172 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:31.549175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:31.549190 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:31.549356 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:31.549396 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.550301 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.550304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:31.551098 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:31.551106 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.551786 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.551790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:31.551893 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:31.551917 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:47:31.552008 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 00:47:31.552084 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":21,"thread_start_us":103,"threads_started":1}
I20260927 00:47:31.552271 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:31.548664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51025 (local address 127.0.98.3:42585)
0927 00:47:31.548831 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:31.548858 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:31.548875 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:47:31.549013 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:31.549016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:31.549067 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:31.549164 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:31.549504 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.550162 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.551229 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.551661 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.551783 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:31.551852 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:31.552039 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 00:47:31.552121 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 00:47:31.552822 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7995f4663704f9f92147fd283f8a510" candidate_uuid: "09c44699d7f34db688c84b5d2e338142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" is_pre_election: true
I20260927 00:47:31.552805 8354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7995f4663704f9f92147fd283f8a510" candidate_uuid: "09c44699d7f34db688c84b5d2e338142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" is_pre_election: true
I20260927 00:47:31.553016 8354 raft_consensus.cc:2454] T b7995f4663704f9f92147fd283f8a510 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09c44699d7f34db688c84b5d2e338142 in term 0.
I20260927 00:47:31.553018 8214 raft_consensus.cc:2454] T b7995f4663704f9f92147fd283f8a510 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09c44699d7f34db688c84b5d2e338142 in term 0.
I20260927 00:47:31.553499 8008 leader_election.cc:302] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c44699d7f34db688c84b5d2e338142, 2f4a4619b3e04e6e8e99a20b640d4650; no voters:
I20260927 00:47:31.553838 8434 raft_consensus.cc:2781] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:47:31.553896 8434 raft_consensus.cc:482] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:47:31.553925 8434 raft_consensus.cc:3037] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:31.555325 8434 raft_consensus.cc:504] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:31.555733 8434 leader_election.cc:288] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142 [CANDIDATE]: Term 1 election: Requested vote from peers a0ae7faedd48405bbc5255ae0acbc6e2 (127.0.98.2:35833), 2f4a4619b3e04e6e8e99a20b640d4650 (127.0.98.3:42585)
I20260927 00:47:31.556164 8354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7995f4663704f9f92147fd283f8a510" candidate_uuid: "09c44699d7f34db688c84b5d2e338142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f4a4619b3e04e6e8e99a20b640d4650"
I20260927 00:47:31.556185 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7995f4663704f9f92147fd283f8a510" candidate_uuid: "09c44699d7f34db688c84b5d2e338142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2"
I20260927 00:47:31.556362 8354 raft_consensus.cc:3037] T b7995f4663704f9f92147fd283f8a510 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:31.556392 8214 raft_consensus.cc:3037] T b7995f4663704f9f92147fd283f8a510 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:31.557478 8354 raft_consensus.cc:2454] T b7995f4663704f9f92147fd283f8a510 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09c44699d7f34db688c84b5d2e338142 in term 1.
I20260927 00:47:31.557554 8214 raft_consensus.cc:2454] T b7995f4663704f9f92147fd283f8a510 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09c44699d7f34db688c84b5d2e338142 in term 1.
I20260927 00:47:31.557849 8008 leader_election.cc:302] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c44699d7f34db688c84b5d2e338142, 2f4a4619b3e04e6e8e99a20b640d4650; no voters:
I20260927 00:47:31.558058 8434 raft_consensus.cc:2781] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:31.558295 8434 raft_consensus.cc:686] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142 [term 1 LEADER]: Becoming Leader. State: Replica: 09c44699d7f34db688c84b5d2e338142, State: Running, Role: LEADER
I20260927 00:47:31.558619 8434 consensus_queue.cc:237] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:31.561599 7914 catalog_manager.cc:5495] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09c44699d7f34db688c84b5d2e338142 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "09c44699d7f34db688c84b5d2e338142" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } health_report { overall_health: UNKNOWN } } }
I20260927 00:47:31.607877 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:31.603629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47856 (local address 127.0.98.1:40447)
0927 00:47:31.603966 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:31.603969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:31.603983 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:47:31.604086 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:31.604088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:31.604164 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:31.604349 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:47:31.604612 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.605202 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.606088 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.606367 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.606455 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:31.606753 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:47:31.607407 (+ 654us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:47:31.607506 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:31.607563 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 00:47:31.607631 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":135,"threads_started":1}
I20260927 00:47:31.611220 8214 raft_consensus.cc:1261] T b7995f4663704f9f92147fd283f8a510 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 1 FOLLOWER]: Refusing update from remote peer 09c44699d7f34db688c84b5d2e338142: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:47:31.611217 8354 raft_consensus.cc:1261] T b7995f4663704f9f92147fd283f8a510 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 1 FOLLOWER]: Refusing update from remote peer 09c44699d7f34db688c84b5d2e338142: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:47:31.611805 8434 consensus_queue.cc:1035] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:47:31.612075 8517 consensus_queue.cc:1035] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:47:31.625239 7914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:47:31.625469 7914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:47:31.628258 7914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7dec1acba204282bf94e1473f15c1bb: Sending DeleteTablet for 3 replicas of tablet b7995f4663704f9f92147fd283f8a510
I20260927 00:47:31.629132 8334 tablet_service.cc:1514] Processing DeleteTablet for tablet b7995f4663704f9f92147fd283f8a510 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42700
I20260927 00:47:31.629106 8194 tablet_service.cc:1514] Processing DeleteTablet for tablet b7995f4663704f9f92147fd283f8a510 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48412
I20260927 00:47:31.629388 8497 tablet_replica.cc:330] T b7995f4663704f9f92147fd283f8a510 P a0ae7faedd48405bbc5255ae0acbc6e2: stopping tablet replica
I20260927 00:47:31.629413 8499 tablet_replica.cc:330] T b7995f4663704f9f92147fd283f8a510 P 2f4a4619b3e04e6e8e99a20b640d4650: stopping tablet replica
I20260927 00:47:31.629556 8499 raft_consensus.cc:2229] T b7995f4663704f9f92147fd283f8a510 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:47:31.629556 8497 raft_consensus.cc:2229] T b7995f4663704f9f92147fd283f8a510 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:47:31.629724 8497 raft_consensus.cc:2258] T b7995f4663704f9f92147fd283f8a510 P a0ae7faedd48405bbc5255ae0acbc6e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:31.629724 8499 raft_consensus.cc:2258] T b7995f4663704f9f92147fd283f8a510 P 2f4a4619b3e04e6e8e99a20b640d4650 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:31.630475 8054 tablet_service.cc:1514] Processing DeleteTablet for tablet b7995f4663704f9f92147fd283f8a510 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34832
I20260927 00:47:31.630564 8497 ts_tablet_manager.cc:1905] T b7995f4663704f9f92147fd283f8a510 P a0ae7faedd48405bbc5255ae0acbc6e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:31.630713 8498 tablet_replica.cc:330] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142: stopping tablet replica
I20260927 00:47:31.630860 8498 raft_consensus.cc:2229] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:47:31.631373 8498 raft_consensus.cc:2258] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:31.631377 7914 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:47:31.632091 7914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:47:31.633987 8497 ts_tablet_manager.cc:1918] T b7995f4663704f9f92147fd283f8a510 P a0ae7faedd48405bbc5255ae0acbc6e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:47:31.634124 8497 log.cc:1196] T b7995f4663704f9f92147fd283f8a510 P a0ae7faedd48405bbc5255ae0acbc6e2: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-1/wal/wals/b7995f4663704f9f92147fd283f8a510
I20260927 00:47:31.634423 8497 ts_tablet_manager.cc:1939] T b7995f4663704f9f92147fd283f8a510 P a0ae7faedd48405bbc5255ae0acbc6e2: Deleting consensus metadata
I20260927 00:47:31.635483 7903 catalog_manager.cc:4841] TS a0ae7faedd48405bbc5255ae0acbc6e2 (127.0.98.2:35833): tablet b7995f4663704f9f92147fd283f8a510 (table test-table [id=b234d69cb4ca479aae1234dfdc3b55f3]) successfully deleted
I20260927 00:47:31.635946 8499 ts_tablet_manager.cc:1905] T b7995f4663704f9f92147fd283f8a510 P 2f4a4619b3e04e6e8e99a20b640d4650: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:31.638796 8334 tablet_service.cc:1467] Processing CreateTablet for tablet 5fcbe35bc26c4aa8b93cac0b5855f7ee (DEFAULT_TABLE table=test-table [id=ab0c0910b0354f879016fd64e752d526]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:31.639106 8499 ts_tablet_manager.cc:1918] T b7995f4663704f9f92147fd283f8a510 P 2f4a4619b3e04e6e8e99a20b640d4650: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:47:31.639217 8499 log.cc:1196] T b7995f4663704f9f92147fd283f8a510 P 2f4a4619b3e04e6e8e99a20b640d4650: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-2/wal/wals/b7995f4663704f9f92147fd283f8a510
I20260927 00:47:31.639406 8054 tablet_service.cc:1467] Processing CreateTablet for tablet 5fcbe35bc26c4aa8b93cac0b5855f7ee (DEFAULT_TABLE table=test-table [id=ab0c0910b0354f879016fd64e752d526]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:31.639497 8499 ts_tablet_manager.cc:1939] T b7995f4663704f9f92147fd283f8a510 P 2f4a4619b3e04e6e8e99a20b640d4650: Deleting consensus metadata
I20260927 00:47:31.639720 8054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fcbe35bc26c4aa8b93cac0b5855f7ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:31.640331 7904 catalog_manager.cc:4841] TS 2f4a4619b3e04e6e8e99a20b640d4650 (127.0.98.3:42585): tablet b7995f4663704f9f92147fd283f8a510 (table test-table [id=b234d69cb4ca479aae1234dfdc3b55f3]) successfully deleted
I20260927 00:47:31.640540 8194 tablet_service.cc:1467] Processing CreateTablet for tablet 5fcbe35bc26c4aa8b93cac0b5855f7ee (DEFAULT_TABLE table=test-table [id=ab0c0910b0354f879016fd64e752d526]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:31.640830 8194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fcbe35bc26c4aa8b93cac0b5855f7ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:31.643563 8429 tablet_bootstrap.cc:492] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P a0ae7faedd48405bbc5255ae0acbc6e2: Bootstrap starting.
I20260927 00:47:31.639117 8334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fcbe35bc26c4aa8b93cac0b5855f7ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:31.644722 8429 tablet_bootstrap.cc:654] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P a0ae7faedd48405bbc5255ae0acbc6e2: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:31.644884 8498 ts_tablet_manager.cc:1905] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:31.646095 8429 tablet_bootstrap.cc:492] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P a0ae7faedd48405bbc5255ae0acbc6e2: No bootstrap required, opened a new log
I20260927 00:47:31.646237 8429 ts_tablet_manager.cc:1397] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P a0ae7faedd48405bbc5255ae0acbc6e2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260927 00:47:31.646771 8429 raft_consensus.cc:348] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:31.646914 8429 raft_consensus.cc:374] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:31.646951 8429 raft_consensus.cc:729] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0ae7faedd48405bbc5255ae0acbc6e2, State: Initialized, Role: FOLLOWER
I20260927 00:47:31.647379 8429 consensus_queue.cc:260] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P a0ae7faedd48405bbc5255ae0acbc6e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:31.647473 8498 ts_tablet_manager.cc:1918] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:47:31.647553 8498 log.cc:1196] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-0/wal/wals/b7995f4663704f9f92147fd283f8a510
I20260927 00:47:31.647822 8498 ts_tablet_manager.cc:1939] T b7995f4663704f9f92147fd283f8a510 P 09c44699d7f34db688c84b5d2e338142: Deleting consensus metadata
I20260927 00:47:31.648343 8429 ts_tablet_manager.cc:1428] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P a0ae7faedd48405bbc5255ae0acbc6e2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 00:47:31.648818 7904 catalog_manager.cc:4841] TS 09c44699d7f34db688c84b5d2e338142 (127.0.98.1:40447): tablet b7995f4663704f9f92147fd283f8a510 (table test-table [id=b234d69cb4ca479aae1234dfdc3b55f3]) successfully deleted
I20260927 00:47:31.649639 8428 tablet_bootstrap.cc:492] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 09c44699d7f34db688c84b5d2e338142: Bootstrap starting.
I20260927 00:47:31.650827 8428 tablet_bootstrap.cc:654] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 09c44699d7f34db688c84b5d2e338142: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:31.652457 8428 tablet_bootstrap.cc:492] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 09c44699d7f34db688c84b5d2e338142: No bootstrap required, opened a new log
I20260927 00:47:31.652580 8428 ts_tablet_manager.cc:1397] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 09c44699d7f34db688c84b5d2e338142: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:47:31.653168 8428 raft_consensus.cc:348] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 09c44699d7f34db688c84b5d2e338142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:31.653326 8428 raft_consensus.cc:374] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 09c44699d7f34db688c84b5d2e338142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:31.653363 8428 raft_consensus.cc:729] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 09c44699d7f34db688c84b5d2e338142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c44699d7f34db688c84b5d2e338142, State: Initialized, Role: FOLLOWER
I20260927 00:47:31.653574 8428 consensus_queue.cc:260] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 09c44699d7f34db688c84b5d2e338142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:31.653883 8428 ts_tablet_manager.cc:1428] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 09c44699d7f34db688c84b5d2e338142: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:47:31.654274 8430 tablet_bootstrap.cc:492] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650: Bootstrap starting.
I20260927 00:47:31.655544 8430 tablet_bootstrap.cc:654] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:31.657052 8430 tablet_bootstrap.cc:492] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650: No bootstrap required, opened a new log
I20260927 00:47:31.657176 8430 ts_tablet_manager.cc:1397] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:47:31.657764 8430 raft_consensus.cc:348] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:31.657923 8430 raft_consensus.cc:374] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:31.657963 8430 raft_consensus.cc:729] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f4a4619b3e04e6e8e99a20b640d4650, State: Initialized, Role: FOLLOWER
I20260927 00:47:31.658151 8430 consensus_queue.cc:260] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:31.658501 8430 ts_tablet_manager.cc:1428] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 00:47:31.896458 8493 raft_consensus.cc:482] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:47:31.896695 8493 raft_consensus.cc:504] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:31.897302 8493 leader_election.cc:288] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09c44699d7f34db688c84b5d2e338142 (127.0.98.1:40447), a0ae7faedd48405bbc5255ae0acbc6e2 (127.0.98.2:35833)
I20260927 00:47:31.897976 8074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcbe35bc26c4aa8b93cac0b5855f7ee" candidate_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09c44699d7f34db688c84b5d2e338142" is_pre_election: true
I20260927 00:47:31.897976 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcbe35bc26c4aa8b93cac0b5855f7ee" candidate_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" is_pre_election: true
I20260927 00:47:31.898188 8074 raft_consensus.cc:2454] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 09c44699d7f34db688c84b5d2e338142 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f4a4619b3e04e6e8e99a20b640d4650 in term 0.
I20260927 00:47:31.898207 8214 raft_consensus.cc:2454] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f4a4619b3e04e6e8e99a20b640d4650 in term 0.
I20260927 00:47:31.898572 8287 leader_election.cc:302] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f4a4619b3e04e6e8e99a20b640d4650, a0ae7faedd48405bbc5255ae0acbc6e2; no voters:
I20260927 00:47:31.898815 8493 raft_consensus.cc:2781] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:47:31.898900 8493 raft_consensus.cc:482] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:47:31.898936 8493 raft_consensus.cc:3037] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:31.900563 8493 raft_consensus.cc:504] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:31.900933 8493 leader_election.cc:288] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650 [CANDIDATE]: Term 1 election: Requested vote from peers 09c44699d7f34db688c84b5d2e338142 (127.0.98.1:40447), a0ae7faedd48405bbc5255ae0acbc6e2 (127.0.98.2:35833)
I20260927 00:47:31.901569 8074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcbe35bc26c4aa8b93cac0b5855f7ee" candidate_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09c44699d7f34db688c84b5d2e338142"
I20260927 00:47:31.901569 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcbe35bc26c4aa8b93cac0b5855f7ee" candidate_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2"
I20260927 00:47:31.901753 8074 raft_consensus.cc:3037] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 09c44699d7f34db688c84b5d2e338142 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:31.901767 8214 raft_consensus.cc:3037] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P a0ae7faedd48405bbc5255ae0acbc6e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:31.903272 8074 raft_consensus.cc:2454] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 09c44699d7f34db688c84b5d2e338142 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f4a4619b3e04e6e8e99a20b640d4650 in term 1.
I20260927 00:47:31.903271 8214 raft_consensus.cc:2454] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P a0ae7faedd48405bbc5255ae0acbc6e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f4a4619b3e04e6e8e99a20b640d4650 in term 1.
I20260927 00:47:31.903748 8288 leader_election.cc:302] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c44699d7f34db688c84b5d2e338142, 2f4a4619b3e04e6e8e99a20b640d4650; no voters:
I20260927 00:47:31.904100 8493 raft_consensus.cc:2781] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:31.904362 8493 raft_consensus.cc:686] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650 [term 1 LEADER]: Becoming Leader. State: Replica: 2f4a4619b3e04e6e8e99a20b640d4650, State: Running, Role: LEADER
I20260927 00:47:31.904559 8493 consensus_queue.cc:237] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } }
I20260927 00:47:31.906656 7917 catalog_manager.cc:5495] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f4a4619b3e04e6e8e99a20b640d4650 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f4a4619b3e04e6e8e99a20b640d4650" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42585 } health_report { overall_health: HEALTHY } } }
I20260927 00:47:31.955015 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:31.951372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41232 (local address 127.0.98.3:42585)
0927 00:47:31.951526 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:31.951533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:31.951548 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:47:31.951717 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:31.951720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:31.951782 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:31.951863 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:47:31.952143 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.952775 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.953731 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.954098 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.954212 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:31.954502 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:47:31.954600 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:47:31.954682 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:31.954776 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 00:47:31.954838 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:47:31.966308 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:31.962835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59465 (local address 127.0.98.3:42585)
0927 00:47:31.962967 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:31.962971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:31.963264 (+ 293us) server_negotiation.cc:407] Connection header received
0927 00:47:31.963330 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:31.963333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:31.963388 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:31.963462 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:47:31.963693 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.964260 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.965218 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.965437 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.965522 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:31.965763 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:47:31.965845 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:47:31.965900 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:31.966081 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 00:47:31.966141 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":51}
I20260927 00:47:31.966322 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:31.962780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42585 (local address 127.0.98.3:59465)
0927 00:47:31.963142 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:31.963155 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:31.963258 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:31.963485 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:31.963488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:31.963496 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:47:31.963597 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:31.963601 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.964382 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.964384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:31.965081 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:31.965090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:31.965559 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:31.965562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:31.965628 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:31.965656 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:47:31.965945 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:47:31.965952 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:47:31.966057 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 00:47:31.966145 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":158,"threads_started":1}
I20260927 00:47:31.978960 8074 raft_consensus.cc:1261] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 09c44699d7f34db688c84b5d2e338142 [term 1 FOLLOWER]: Refusing update from remote peer 2f4a4619b3e04e6e8e99a20b640d4650: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:47:31.978960 8214 raft_consensus.cc:1261] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P a0ae7faedd48405bbc5255ae0acbc6e2 [term 1 FOLLOWER]: Refusing update from remote peer 2f4a4619b3e04e6e8e99a20b640d4650: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:47:31.979641 8529 consensus_queue.cc:1035] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09c44699d7f34db688c84b5d2e338142" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:47:31.980012 8527 consensus_queue.cc:1035] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0ae7faedd48405bbc5255ae0acbc6e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:47:32.023546 7914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:47:32.023815 7914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:47:32.026749 7914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7dec1acba204282bf94e1473f15c1bb: Sending DeleteTablet for 3 replicas of tablet 5fcbe35bc26c4aa8b93cac0b5855f7ee
I20260927 00:47:32.027650 8194 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fcbe35bc26c4aa8b93cac0b5855f7ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48412
I20260927 00:47:32.027694 8054 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fcbe35bc26c4aa8b93cac0b5855f7ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34832
I20260927 00:47:32.027905 8497 tablet_replica.cc:330] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P a0ae7faedd48405bbc5255ae0acbc6e2: stopping tablet replica
I20260927 00:47:32.027941 8498 tablet_replica.cc:330] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 09c44699d7f34db688c84b5d2e338142: stopping tablet replica
I20260927 00:47:32.028077 8498 raft_consensus.cc:2229] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 09c44699d7f34db688c84b5d2e338142 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:47:32.028084 8497 raft_consensus.cc:2229] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P a0ae7faedd48405bbc5255ae0acbc6e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:47:32.028262 8498 raft_consensus.cc:2258] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 09c44699d7f34db688c84b5d2e338142 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:32.028262 8497 raft_consensus.cc:2258] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P a0ae7faedd48405bbc5255ae0acbc6e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:32.028738 8334 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fcbe35bc26c4aa8b93cac0b5855f7ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42700
I20260927 00:47:32.028966 8499 tablet_replica.cc:330] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650: stopping tablet replica
I20260927 00:47:32.029125 8499 raft_consensus.cc:2229] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:47:32.029297 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 7989
I20260927 00:47:32.029572 8499 raft_consensus.cc:2258] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:32.029600 8498 ts_tablet_manager.cc:1905] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 09c44699d7f34db688c84b5d2e338142: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:32.033308 8499 ts_tablet_manager.cc:1905] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:32.037164 8497 ts_tablet_manager.cc:1905] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P a0ae7faedd48405bbc5255ae0acbc6e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:32.037902 8499 ts_tablet_manager.cc:1918] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:47:32.038018 8499 log.cc:1196] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-2/wal/wals/5fcbe35bc26c4aa8b93cac0b5855f7ee
I20260927 00:47:32.038363 8499 ts_tablet_manager.cc:1939] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P 2f4a4619b3e04e6e8e99a20b640d4650: Deleting consensus metadata
I20260927 00:47:32.040387 8497 ts_tablet_manager.cc:1918] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P a0ae7faedd48405bbc5255ae0acbc6e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:47:32.040498 8497 log.cc:1196] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P a0ae7faedd48405bbc5255ae0acbc6e2: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790470006085796-392-0/minicluster-data/ts-1/wal/wals/5fcbe35bc26c4aa8b93cac0b5855f7ee
I20260927 00:47:32.040781 8497 ts_tablet_manager.cc:1939] T 5fcbe35bc26c4aa8b93cac0b5855f7ee P a0ae7faedd48405bbc5255ae0acbc6e2: Deleting consensus metadata
I20260927 00:47:32.041847 7904 catalog_manager.cc:4841] TS 2f4a4619b3e04e6e8e99a20b640d4650 (127.0.98.3:42585): tablet 5fcbe35bc26c4aa8b93cac0b5855f7ee (table test-table [id=ab0c0910b0354f879016fd64e752d526]) successfully deleted
I20260927 00:47:32.041847 7903 catalog_manager.cc:4841] TS a0ae7faedd48405bbc5255ae0acbc6e2 (127.0.98.2:35833): tablet 5fcbe35bc26c4aa8b93cac0b5855f7ee (table test-table [id=ab0c0910b0354f879016fd64e752d526]) successfully deleted
W20260927 00:47:32.043167 7904 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40447 (error 108)
I20260927 00:47:32.043252 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 8129
I20260927 00:47:32.045818 8540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:32.045237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40447 (local address 127.0.0.1:47858)
0927 00:47:32.045548 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:32.045653 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":81,"threads_started":1}
W20260927 00:47:32.045974 7904 catalog_manager.cc:4584] TS 09c44699d7f34db688c84b5d2e338142 (127.0.98.1:40447): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5fcbe35bc26c4aa8b93cac0b5855f7ee: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40447: connect: Connection refused (error 111)
I20260927 00:47:32.053149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 8269
I20260927 00:47:32.061630 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 7885
2026-09-27T00:47:32Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (4146 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:47:32 dist-test-slave-86rs krb5kdc[8543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:47:32 dist-test-slave-86rs krb5kdc[8543](info): set up 2 sockets
Sep 27 00:47:32 dist-test-slave-86rs krb5kdc[8543](info): commencing operation
krb5kdc: starting...
W20260927 00:47:33.149711 392 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:47:33 dist-test-slave-86rs krb5kdc[8543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:47:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:47:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:47:33.292412 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40809
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:33.430775 8556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:33.431120 8556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:33.431173 8556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:33.435432 8556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:47:33.435518 8556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:33.435539 8556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:33.435556 8556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:47:33.435573 8556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:47:33.441558 8556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:33.443058 8556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:33.444465 8556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:33.454994 8562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:33.454994 8564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:33.455178 8561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:33.455222 8556 server_base.cc:1034] running on GCE node
I20260927 00:47:33.456460 8556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:33.458097 8556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:33.459287 8556 hybrid_clock.cc:648] HybridClock initialized: now 1790470053459268 us; error 38 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T00:47:33Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1408 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:47:33 dist-test-slave-86rs krb5kdc[8569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:47:33 dist-test-slave-86rs krb5kdc[8569](info): set up 2 sockets
Sep 27 00:47:33 dist-test-slave-86rs krb5kdc[8569](info): commencing operation
krb5kdc: starting...
W20260927 00:47:34.549608 392 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:47:34 dist-test-slave-86rs krb5kdc[8569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:47:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:47:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:47:34.689142 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34583
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:34.826239 8582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:34.826704 8582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:34.826794 8582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:34.831539 8582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:47:34.831696 8582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:34.831735 8582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:34.831768 8582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:47:34.831804 8582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:47:34.838369 8582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:34.839986 8582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:34.841496 8582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:34.852052 8588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:34.852062 8587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:34.852052 8590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:34.852479 8582 server_base.cc:1034] running on GCE node
I20260927 00:47:34.853199 8582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:34.854789 8582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:34.856003 8582 hybrid_clock.cc:648] HybridClock initialized: now 1790470054855968 us; error 54 us; skew 500 ppm
Sep 27 00:47:34 dist-test-slave-86rs krb5kdc[8569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:34.859639 8582 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:47:34.861088 8582 webserver.cc:460] Webserver started at http://127.0.98.62:42871/ using document root <none> and password file <none>
I20260927 00:47:34.862040 8582 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:34.862124 8582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:34.862434 8582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:34.865505 8582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "38ba47a5cc694cb0bf467832914ffa3f"
format_stamp: "Formatted at 2026-09-27 00:47:34 on dist-test-slave-86rs"
server_key: "b3404f70ecd927621c3062b32e096011"
server_key_iv: "7cc658cbc0b534aba2436f8a0e46ef96"
server_key_version: "encryptionkey@0"
I20260927 00:47:34.866328 8582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "38ba47a5cc694cb0bf467832914ffa3f"
format_stamp: "Formatted at 2026-09-27 00:47:34 on dist-test-slave-86rs"
server_key: "b3404f70ecd927621c3062b32e096011"
server_key_iv: "7cc658cbc0b534aba2436f8a0e46ef96"
server_key_version: "encryptionkey@0"
I20260927 00:47:34.871464 8582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 00:47:34.874716 8597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:34.876693 8582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 00:47:34.876884 8582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "38ba47a5cc694cb0bf467832914ffa3f"
format_stamp: "Formatted at 2026-09-27 00:47:34 on dist-test-slave-86rs"
server_key: "b3404f70ecd927621c3062b32e096011"
server_key_iv: "7cc658cbc0b534aba2436f8a0e46ef96"
server_key_version: "encryptionkey@0"
I20260927 00:47:34.877079 8582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:34.916288 8582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:34.920651 8582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:34.920914 8582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:34.930172 8582 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34583
I20260927 00:47:34.930176 8649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34583 every 8 connection(s)
I20260927 00:47:34.931557 8582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:47:34.935284 8650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:34.938511 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 8582
I20260927 00:47:34.938694 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:47:34.939057 392 external_mini_cluster.cc:1428] Setting key 996a655ac6f30d48361a489904234a3b
I20260927 00:47:34.944440 8650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f: Bootstrap starting.
I20260927 00:47:34.947979 8650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:34.948974 8650 log.cc:824] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:47:34 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:34.951547 8650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f: No bootstrap required, opened a new log
I20260927 00:47:34.955729 8650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ba47a5cc694cb0bf467832914ffa3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34583 } }
I20260927 00:47:34.956036 8650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:34.956116 8650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ba47a5cc694cb0bf467832914ffa3f, State: Initialized, Role: FOLLOWER
I20260927 00:47:34.956385 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:34.941318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50738 (local address 127.0.98.62:34583)
0927 00:47:34.941902 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:34.941922 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:34.941978 (+ 56us) server_negotiation.cc:407] Connection header received
0927 00:47:34.942937 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:34.942969 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:34.943484 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:34.943890 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:34.945014 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:34.946178 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:34.947503 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:34.948081 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:34.948299 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:34.951141 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:34.951147 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:34.951166 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:34.951197 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:34.953059 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:34.953538 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:34.953545 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:34.953551 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:34.953682 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:34.953966 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:34.953974 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:34.953975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:34.954446 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:34.954693 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:34.954963 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 00:47:34.955312 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":179,"threads_started":1}
I20260927 00:47:34.956801 8650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ba47a5cc694cb0bf467832914ffa3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34583 } }
I20260927 00:47:34.956992 8650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:47:34.957058 8650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:47:34.957168 8650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:34.958388 8650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ba47a5cc694cb0bf467832914ffa3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34583 } }
I20260927 00:47:34.958810 8650 leader_election.cc:302] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38ba47a5cc694cb0bf467832914ffa3f; no voters:
I20260927 00:47:34.959257 8650 leader_election.cc:288] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:47:34.959381 8655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:34.959661 8655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [term 1 LEADER]: Becoming Leader. State: Replica: 38ba47a5cc694cb0bf467832914ffa3f, State: Running, Role: LEADER
I20260927 00:47:34.960084 8655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ba47a5cc694cb0bf467832914ffa3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34583 } }
I20260927 00:47:34.960857 8650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:47:34.962255 8657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38ba47a5cc694cb0bf467832914ffa3f. Latest consensus state: current_term: 1 leader_uuid: "38ba47a5cc694cb0bf467832914ffa3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ba47a5cc694cb0bf467832914ffa3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34583 } } }
I20260927 00:47:34.962433 8657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:34.962898 8664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:47:34.962857 8656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38ba47a5cc694cb0bf467832914ffa3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ba47a5cc694cb0bf467832914ffa3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34583 } } }
I20260927 00:47:34.962990 8656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:34.966647 8664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:47:34.973960 8664 catalog_manager.cc:1347] Generated new cluster ID: c24960be2f004f4aa2ab5b2c4ca40f1a
I20260927 00:47:34.974062 8664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:47:34.987561 8664 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:47:34.988915 8664 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:47:35.002810 8664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f: Generated new TSK 0
I20260927 00:47:35.003854 8664 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:47:35.047418 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34583
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:35.200467 8676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:35.200861 8676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:35.200913 8676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:35.205574 8676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:35.205650 8676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:35.205719 8676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:47:35.211745 8676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:35.213330 8676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:35.214705 8676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:47:35.226962 8676 server_base.cc:1034] running on GCE node
W20260927 00:47:35.226828 8681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:35.226925 8682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:35.226828 8684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:35.227799 8676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:35.228770 8676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:35.229996 8676 hybrid_clock.cc:648] HybridClock initialized: now 1790470055229956 us; error 49 us; skew 500 ppm
Sep 27 00:47:35 dist-test-slave-86rs krb5kdc[8569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:35.234093 8676 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:47:35.235479 8676 webserver.cc:460] Webserver started at http://127.0.98.1:43465/ using document root <none> and password file <none>
I20260927 00:47:35.236341 8676 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:35.236454 8676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:35.236748 8676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:35.239478 8676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/data/instance:
uuid: "33c72d34fad740a78229d249ee0f9239"
format_stamp: "Formatted at 2026-09-27 00:47:35 on dist-test-slave-86rs"
server_key: "64d9611e7324413ff3ff4909ba44674c"
server_key_iv: "730a0098c33b82334b11e47d132c4c93"
server_key_version: "encryptionkey@0"
I20260927 00:47:35.240188 8676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/wal/instance:
uuid: "33c72d34fad740a78229d249ee0f9239"
format_stamp: "Formatted at 2026-09-27 00:47:35 on dist-test-slave-86rs"
server_key: "64d9611e7324413ff3ff4909ba44674c"
server_key_iv: "730a0098c33b82334b11e47d132c4c93"
server_key_version: "encryptionkey@0"
I20260927 00:47:35.245888 8676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 00:47:35.249272 8691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:35.251039 8676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:47:35.251201 8676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "33c72d34fad740a78229d249ee0f9239"
format_stamp: "Formatted at 2026-09-27 00:47:35 on dist-test-slave-86rs"
server_key: "64d9611e7324413ff3ff4909ba44674c"
server_key_iv: "730a0098c33b82334b11e47d132c4c93"
server_key_version: "encryptionkey@0"
I20260927 00:47:35.251335 8676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:35.272892 8676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:35.277284 8676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:35.277494 8676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:35.278220 8676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:35.279383 8676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:35.279438 8676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:35.279485 8676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:35.279506 8676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:35.290130 8676 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33403
I20260927 00:47:35.290154 8804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33403 every 8 connection(s)
I20260927 00:47:35.291340 8676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:47:35.297206 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 8676
I20260927 00:47:35.297410 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:47:35.297787 392 external_mini_cluster.cc:1428] Setting key 4ef34b34590e6b15d9d56323906e4d66
Sep 27 00:47:35 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:47:35.310695 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:35.293778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44627 (local address 127.0.98.62:34583)
0927 00:47:35.293988 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:35.293994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:35.295284 (+ 1290us) server_negotiation.cc:407] Connection header received
0927 00:47:35.297321 (+ 2037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:35.297325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:35.297413 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:35.297529 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:35.299576 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:35.300400 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:35.301842 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:35.302159 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:35.302253 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:35.305937 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:35.305941 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:35.305943 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:35.305974 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:35.307656 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:35.308537 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:35.308540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:35.308542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:35.308643 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:35.309154 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:35.309157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:35.309158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:35.309415 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:35.309499 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:35.310323 (+ 824us) server_negotiation.cc:299] Negotiation successful
0927 00:47:35.310483 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 00:47:35.311900 8807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:35.294401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.1:44627)
0927 00:47:35.295061 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:35.295120 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:35.297025 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:35.297772 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:35.297782 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:35.298364 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:35.299239 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:35.299268 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:35.300541 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:35.300544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:35.301650 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:35.301682 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:35.302349 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:35.302355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:35.302624 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:35.303473 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:35.303493 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:35.305702 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:35.307808 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:35.307818 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:35.307849 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:35.308382 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:35.308770 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:35.308775 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:35.308778 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:35.309039 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:35.309526 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:35.309538 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:35.310011 (+ 473us) client_negotiation.cc:769] Sending connection context
0927 00:47:35.310218 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 00:47:35.310645 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":171,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:47:35.313606 8805 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34583
I20260927 00:47:35.313979 8805 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:35.314754 8805 heartbeater.cc:511] Master 127.0.98.62:34583 requested a full tablet report, sending...
I20260927 00:47:35.316847 8614 ts_manager.cc:194] Registered new tserver with Master: 33c72d34fad740a78229d249ee0f9239 (127.0.98.1:33403)
I20260927 00:47:35.318218 8614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:47:35.329782 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34583
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:35.471438 8810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:35.471832 8810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:35.471910 8810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:35.476516 8810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:35.476625 8810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:35.476747 8810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:47:35.483278 8810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:35.485075 8810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:35.486501 8810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:35.498929 8816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:35.498975 8818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:35.498930 8815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:35.499046 8810 server_base.cc:1034] running on GCE node
I20260927 00:47:35.499766 8810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:35.500697 8810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:35.501901 8810 hybrid_clock.cc:648] HybridClock initialized: now 1790470055501868 us; error 51 us; skew 500 ppm
Sep 27 00:47:35 dist-test-slave-86rs krb5kdc[8569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:35.505834 8810 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:47:35.507201 8810 webserver.cc:460] Webserver started at http://127.0.98.2:43393/ using document root <none> and password file <none>
I20260927 00:47:35.508064 8810 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:35.508195 8810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:35.508546 8810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:35.511497 8810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/data/instance:
uuid: "a70629b34efd4a059c07b2c6e3c1a976"
format_stamp: "Formatted at 2026-09-27 00:47:35 on dist-test-slave-86rs"
server_key: "e927f1a973289083fb45a7d356414bde"
server_key_iv: "3fcadab41cd2cff4f4c4302bfa8ffa2d"
server_key_version: "encryptionkey@0"
I20260927 00:47:35.512377 8810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a70629b34efd4a059c07b2c6e3c1a976"
format_stamp: "Formatted at 2026-09-27 00:47:35 on dist-test-slave-86rs"
server_key: "e927f1a973289083fb45a7d356414bde"
server_key_iv: "3fcadab41cd2cff4f4c4302bfa8ffa2d"
server_key_version: "encryptionkey@0"
I20260927 00:47:35.517453 8810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260927 00:47:35.520742 8825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:35.522418 8810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:47:35.522578 8810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "a70629b34efd4a059c07b2c6e3c1a976"
format_stamp: "Formatted at 2026-09-27 00:47:35 on dist-test-slave-86rs"
server_key: "e927f1a973289083fb45a7d356414bde"
server_key_iv: "3fcadab41cd2cff4f4c4302bfa8ffa2d"
server_key_version: "encryptionkey@0"
I20260927 00:47:35.522725 8810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:35.548637 8810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:35.552953 8810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:35.553210 8810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:35.554004 8810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:35.555124 8810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:35.555209 8810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:35.555289 8810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:35.555334 8810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:35.566506 8810 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36611
I20260927 00:47:35.566525 8938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36611 every 8 connection(s)
I20260927 00:47:35.567744 8810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:47:35.568897 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 8810
I20260927 00:47:35.569042 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:47:35.569322 392 external_mini_cluster.cc:1428] Setting key c30ddb835902baa9d16f8df97c6b61f4
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 27 00:47:35 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:47:35.585740 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:35.570264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43263 (local address 127.0.98.62:34583)
0927 00:47:35.570430 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:35.570435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:35.571870 (+ 1435us) server_negotiation.cc:407] Connection header received
0927 00:47:35.573648 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:35.573653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:35.573717 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:35.573824 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:35.575379 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:35.576041 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:35.577457 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:35.577699 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:35.577819 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:35.580980 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:35.580985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:35.580987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:35.581021 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:35.582780 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:35.583587 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:35.583591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:35.583593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:35.583699 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:35.584132 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:35.584136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:35.584137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:35.584471 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:35.584598 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:35.585349 (+ 751us) server_negotiation.cc:299] Negotiation successful
0927 00:47:35.585521 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:47:35.587028 8941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:35.570976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.2:43263)
0927 00:47:35.571686 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:35.571740 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:35.573383 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:35.574069 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:35.574079 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:35.574554 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:35.575206 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:35.575224 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:35.576199 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:35.576202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:35.577307 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:35.577317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:35.577880 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:35.577885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:35.578095 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:35.578897 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:35.578917 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:35.580721 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:35.582960 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:35.582970 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:35.583002 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:35.583450 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:35.583812 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:35.583815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:35.583817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:35.584020 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:35.584598 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:35.584608 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:35.585029 (+ 421us) client_negotiation.cc:769] Sending connection context
0927 00:47:35.585296 (+ 267us) client_negotiation.cc:240] Negotiation successful
0927 00:47:35.585791 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":212,"threads_started":1}
I20260927 00:47:35.588847 8939 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34583
I20260927 00:47:35.589236 8939 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:35.590075 8939 heartbeater.cc:511] Master 127.0.98.62:34583 requested a full tablet report, sending...
I20260927 00:47:35.591673 8614 ts_manager.cc:194] Registered new tserver with Master: a70629b34efd4a059c07b2c6e3c1a976 (127.0.98.2:36611)
I20260927 00:47:35.592765 8614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:47:35.607020 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34583
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:35.760573 8944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:35.761065 8944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:35.761183 8944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:35.767112 8944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:35.767235 8944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:35.767357 8944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:47:35.775024 8944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:35.776769 8944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:35.778312 8944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:35.792181 8952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:35.792181 8950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:35.792405 8949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:35.792579 8944 server_base.cc:1034] running on GCE node
I20260927 00:47:35.793273 8944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:35.794236 8944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:35.795450 8944 hybrid_clock.cc:648] HybridClock initialized: now 1790470055795420 us; error 46 us; skew 500 ppm
Sep 27 00:47:35 dist-test-slave-86rs krb5kdc[8569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:35.799238 8944 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:47:35.800619 8944 webserver.cc:460] Webserver started at http://127.0.98.3:38291/ using document root <none> and password file <none>
I20260927 00:47:35.801411 8944 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:35.801512 8944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:35.801805 8944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:35.804816 8944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/data/instance:
uuid: "cb1e88594ee14a7097ee9a8cac788f41"
format_stamp: "Formatted at 2026-09-27 00:47:35 on dist-test-slave-86rs"
server_key: "edd0d0e9ec3c5f5682f6b24b08b54e9d"
server_key_iv: "8aee9bf3b3a71718e9d37b2ca7364813"
server_key_version: "encryptionkey@0"
I20260927 00:47:35.805604 8944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cb1e88594ee14a7097ee9a8cac788f41"
format_stamp: "Formatted at 2026-09-27 00:47:35 on dist-test-slave-86rs"
server_key: "edd0d0e9ec3c5f5682f6b24b08b54e9d"
server_key_iv: "8aee9bf3b3a71718e9d37b2ca7364813"
server_key_version: "encryptionkey@0"
I20260927 00:47:35.810916 8944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 00:47:35.814448 8959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:35.816296 8944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:47:35.816462 8944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "cb1e88594ee14a7097ee9a8cac788f41"
format_stamp: "Formatted at 2026-09-27 00:47:35 on dist-test-slave-86rs"
server_key: "edd0d0e9ec3c5f5682f6b24b08b54e9d"
server_key_iv: "8aee9bf3b3a71718e9d37b2ca7364813"
server_key_version: "encryptionkey@0"
I20260927 00:47:35.816618 8944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:35.835706 8944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:35.840154 8944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:35.840471 8944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:35.841320 8944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:35.842471 8944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:35.842562 8944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:35.842669 8944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:35.842720 8944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:35.853963 8944 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37143
I20260927 00:47:35.853988 9072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37143 every 8 connection(s)
I20260927 00:47:35.855335 8944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:47:35.856734 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 8944
I20260927 00:47:35.856904 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:47:35.857308 392 external_mini_cluster.cc:1428] Setting key c7fafac3c616757ca8dc9861229f64b7
Sep 27 00:47:35 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:35.874982 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:35.858318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50017 (local address 127.0.98.62:34583)
0927 00:47:35.858560 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:35.858565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:35.860444 (+ 1879us) server_negotiation.cc:407] Connection header received
0927 00:47:35.862360 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:35.862378 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:35.862520 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:35.862619 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:35.864456 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:35.865115 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:35.866596 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:35.866877 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:35.867014 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:35.870456 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:35.870461 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:35.870463 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:35.870500 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:35.872660 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:35.873291 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:35.873295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:35.873296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:35.873385 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:35.873767 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:35.873769 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:35.873771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:35.874003 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:35.874090 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:35.874706 (+ 616us) server_negotiation.cc:299] Negotiation successful
0927 00:47:35.874828 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 00:47:35.876252 9075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:35.858819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.3:50017)
0927 00:47:35.860214 (+ 1395us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:35.860336 (+ 122us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:35.862067 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:35.862877 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:35.862888 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:35.863415 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:35.864218 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:35.864306 (+ 88us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:35.865282 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:35.865285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:35.866434 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:35.866445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:35.867082 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:35.867086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:35.867306 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:35.868148 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:35.868171 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:35.870177 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:35.872796 (+ 2619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:35.872808 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:35.872839 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:35.873187 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:35.873488 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:35.873491 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:35.873493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:35.873665 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:35.874094 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:35.874100 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:35.874445 (+ 345us) client_negotiation.cc:769] Sending connection context
0927 00:47:35.874648 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 00:47:35.875007 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1059,"thread_start_us":209,"threads_started":1}
I20260927 00:47:35.877846 9073 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34583
I20260927 00:47:35.878213 9073 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:35.878898 9073 heartbeater.cc:511] Master 127.0.98.62:34583 requested a full tablet report, sending...
I20260927 00:47:35.880339 8614 ts_manager.cc:194] Registered new tserver with Master: cb1e88594ee14a7097ee9a8cac788f41 (127.0.98.3:37143)
I20260927 00:47:35.881105 8614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50017
I20260927 00:47:35.884850 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 00:47:35.975919 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 8676
I20260927 00:47:35.987330 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 8810
I20260927 00:47:35.994189 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 8944
I20260927 00:47:36.000691 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 8582
I20260927 00:47:36.008702 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34583
--webserver_interface=127.0.98.62
--webserver_port=42871
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:36.149679 9082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:36.150722 9082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:36.150815 9082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:36.155812 9082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:47:36.155941 9082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:36.155963 9082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:36.155983 9082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:47:36.156000 9082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:47:36.164093 9082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:36.165958 9082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:36.167644 9082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:36.179337 9090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:36.179337 9087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:36.179617 9082 server_base.cc:1034] running on GCE node
W20260927 00:47:36.179337 9088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:36.180655 9082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:36.182291 9082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:36.183548 9082 hybrid_clock.cc:648] HybridClock initialized: now 1790470056183523 us; error 48 us; skew 500 ppm
Sep 27 00:47:36 dist-test-slave-86rs krb5kdc[8569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:36.187707 9082 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:47:36.189157 9082 webserver.cc:460] Webserver started at http://127.0.98.62:42871/ using document root <none> and password file <none>
I20260927 00:47:36.189942 9082 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:36.190074 9082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:36.195428 9082 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:47:36.198382 9097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:36.200073 9082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:47:36.200311 9082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "38ba47a5cc694cb0bf467832914ffa3f"
format_stamp: "Formatted at 2026-09-27 00:47:34 on dist-test-slave-86rs"
server_key: "b3404f70ecd927621c3062b32e096011"
server_key_iv: "7cc658cbc0b534aba2436f8a0e46ef96"
server_key_version: "encryptionkey@0"
I20260927 00:47:36.201045 9082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:36.241545 9082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:36.245741 9082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:36.245993 9082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:36.255195 9082 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34583
I20260927 00:47:36.255209 9149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34583 every 8 connection(s)
I20260927 00:47:36.256698 9082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:47:36.258179 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 9082
I20260927 00:47:36.258700 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33403
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34583
--webserver_port=43465
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:47:36.262663 9150 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:47:36.265049 9150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f: Bootstrap starting.
I20260927 00:47:36.285635 9150 log.cc:824] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:36.293617 9150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:47:36.294090 9150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f: Bootstrap complete.
I20260927 00:47:36.298978 9150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ba47a5cc694cb0bf467832914ffa3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34583 } }
I20260927 00:47:36.299482 9150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ba47a5cc694cb0bf467832914ffa3f, State: Initialized, Role: FOLLOWER
I20260927 00:47:36.300103 9150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ba47a5cc694cb0bf467832914ffa3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34583 } }
I20260927 00:47:36.300276 9150 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:47:36.300369 9150 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:47:36.300496 9150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:47:36.302800 9150 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ba47a5cc694cb0bf467832914ffa3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34583 } }
I20260927 00:47:36.303190 9150 leader_election.cc:302] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38ba47a5cc694cb0bf467832914ffa3f; no voters:
I20260927 00:47:36.303614 9150 leader_election.cc:288] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 00:47:36.303719 9154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [term 2 FOLLOWER]: Leader election won for term 2
I20260927 00:47:36.303932 9154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [term 2 LEADER]: Becoming Leader. State: Replica: 38ba47a5cc694cb0bf467832914ffa3f, State: Running, Role: LEADER
I20260927 00:47:36.304373 9154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ba47a5cc694cb0bf467832914ffa3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34583 } }
I20260927 00:47:36.305176 9150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:47:36.306041 9155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "38ba47a5cc694cb0bf467832914ffa3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ba47a5cc694cb0bf467832914ffa3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34583 } } }
I20260927 00:47:36.306088 9156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38ba47a5cc694cb0bf467832914ffa3f. Latest consensus state: current_term: 2 leader_uuid: "38ba47a5cc694cb0bf467832914ffa3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ba47a5cc694cb0bf467832914ffa3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34583 } } }
I20260927 00:47:36.306183 9155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:36.306209 9156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:36.307329 9161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:47:36.311955 9161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:47:36.312734 9161 catalog_manager.cc:1259] Loaded cluster ID: c24960be2f004f4aa2ab5b2c4ca40f1a
I20260927 00:47:36.312831 9161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:47:36.315387 9161 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:47:36.316154 9161 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 38ba47a5cc694cb0bf467832914ffa3f: Loaded TSK: 0
I20260927 00:47:36.317682 9161 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:47:36.399732 9152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:36.400133 9152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:36.400189 9152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:36.405074 9152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:36.405164 9152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:36.405267 9152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:47:36.411586 9152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43465
--tserver_master_addrs=127.0.98.62:34583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:36.413147 9152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:36.414563 9152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:36.428215 9177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:36.428256 9179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:36.428246 9176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:36.428488 9152 server_base.cc:1034] running on GCE node
I20260927 00:47:36.429149 9152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:36.430045 9152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:36.431241 9152 hybrid_clock.cc:648] HybridClock initialized: now 1790470056431228 us; error 38 us; skew 500 ppm
Sep 27 00:47:36 dist-test-slave-86rs krb5kdc[8569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470056, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:36.434502 9152 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 00:47:36.435804 9152 webserver.cc:460] Webserver started at http://127.0.98.1:43465/ using document root <none> and password file <none>
I20260927 00:47:36.436604 9152 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:36.436702 9152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:36.442507 9152 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:47:36.445436 9186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:36.447183 9152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:47:36.447338 9152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "33c72d34fad740a78229d249ee0f9239"
format_stamp: "Formatted at 2026-09-27 00:47:35 on dist-test-slave-86rs"
server_key: "64d9611e7324413ff3ff4909ba44674c"
server_key_iv: "730a0098c33b82334b11e47d132c4c93"
server_key_version: "encryptionkey@0"
I20260927 00:47:36.448055 9152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:36.461740 9152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:36.465646 9152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:36.465885 9152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:36.466714 9152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:36.467759 9152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:36.467809 9152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:36.467880 9152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:36.467911 9152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:36.479074 9152 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33403
I20260927 00:47:36.479090 9299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33403 every 8 connection(s)
I20260927 00:47:36.480592 9152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:47:36.487238 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 9152
I20260927 00:47:36.487921 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36611
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34583
--webserver_port=43393
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:47:36 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:36 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 00:47:36.499519 9302 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:36.483543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58073 (local address 127.0.98.62:34583)
0927 00:47:36.484190 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:36.484217 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:36.484580 (+ 363us) server_negotiation.cc:407] Connection header received
0927 00:47:36.486582 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:36.486597 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:36.487160 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:36.487698 (+ 538us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:36.489336 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.490641 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.492168 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.493240 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.493512 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:36.498499 (+ 4987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58073: BlockingRecv error: recv got EOF from 127.0.98.1:58073 (error 108)
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":131,"threads_started":1}
I20260927 00:47:36.499533 9303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:36.483702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.1:58073)
0927 00:47:36.484376 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:36.484434 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:36.486030 (+ 1596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:36.487781 (+ 1751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:36.487789 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:36.488419 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:36.489105 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:36.489127 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.490846 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.490849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:36.492006 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:36.492020 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.493497 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.493502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:36.493706 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:36.494537 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:36.494559 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:36.498153 (+ 3594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":178,"threads_started":1}
W20260927 00:47:36.500277 9300 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34583 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34583: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 00:47:36 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:36 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:36.507300 9303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:36.500905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.1:50647)
0927 00:47:36.501088 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:36.501109 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:36.501221 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:36.501494 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:36.501497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:36.501700 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:36.501840 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:36.501846 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.502697 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.502701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:36.503652 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:36.503663 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.504090 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.504094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:36.504189 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:36.504764 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:36.504779 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:36.507097 (+ 2318us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 00:47:36.507378 9302 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:36.501035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50647 (local address 127.0.98.62:34583)
0927 00:47:36.501162 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:36.501165 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:36.501179 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:47:36.501326 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:36.501328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:36.501380 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:36.501454 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:36.501957 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.502536 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.503762 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.503988 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.504074 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:36.507253 (+ 3179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50647: BlockingRecv error: recv got EOF from 127.0.98.1:50647 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 00:47:36 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:36 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:36.513949 9303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:36.508039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.1:56031)
0927 00:47:36.508146 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:36.508163 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:36.508289 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:36.508550 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:36.508552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:36.508700 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:36.508786 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:36.508791 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.509839 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.509842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:36.510398 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:36.510406 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.510990 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.510993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:36.511064 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:36.511657 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:36.511669 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:36.513810 (+ 2141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 00:47:36.514232 9300 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:47:36.514467 9302 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:36.508112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56031 (local address 127.0.98.62:34583)
0927 00:47:36.508259 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:36.508263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:36.508277 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:47:36.508386 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:36.508389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:36.508440 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:36.508533 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:36.508882 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.509733 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.510512 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.510873 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.510974 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:36.514020 (+ 3046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56031: BlockingRecv error: recv got EOF from 127.0.98.1:56031 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 00:47:36.632474 9304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:36.632848 9304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:36.632899 9304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:36.637259 9304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:36.637346 9304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:36.637416 9304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:47:36.643822 9304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36611
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43393
--tserver_master_addrs=127.0.98.62:34583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:36.645488 9304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:36.646927 9304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:47:36.660141 9304 server_base.cc:1034] running on GCE node
W20260927 00:47:36.660128 9310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:36.660068 9309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:36.660068 9312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:36.661027 9304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:36.662001 9304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:36.663208 9304 hybrid_clock.cc:648] HybridClock initialized: now 1790470056663184 us; error 44 us; skew 500 ppm
Sep 27 00:47:36 dist-test-slave-86rs krb5kdc[8569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470056, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:36.666890 9304 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 00:47:36.668087 9304 webserver.cc:460] Webserver started at http://127.0.98.2:43393/ using document root <none> and password file <none>
I20260927 00:47:36.668871 9304 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:36.668956 9304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:36.674091 9304 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:47:36.677028 9319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:36.678668 9304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:47:36.678853 9304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "a70629b34efd4a059c07b2c6e3c1a976"
format_stamp: "Formatted at 2026-09-27 00:47:35 on dist-test-slave-86rs"
server_key: "e927f1a973289083fb45a7d356414bde"
server_key_iv: "3fcadab41cd2cff4f4c4302bfa8ffa2d"
server_key_version: "encryptionkey@0"
I20260927 00:47:36.679558 9304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:36.700470 9304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:36.705056 9304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:36.705320 9304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:36.706310 9304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:36.707453 9304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:36.707538 9304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:36.707625 9304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:36.707665 9304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:36.720194 9304 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36611
I20260927 00:47:36.720211 9432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36611 every 8 connection(s)
I20260927 00:47:36.721710 9304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:47:36.727533 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 9304
I20260927 00:47:36.728089 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37143
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34583
--webserver_port=38291
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:47:36 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:36 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 00:47:36.736413 9302 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:36.724194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56925 (local address 127.0.98.62:34583)
0927 00:47:36.724394 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:36.724399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:36.725590 (+ 1191us) server_negotiation.cc:407] Connection header received
0927 00:47:36.727206 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:36.727210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:36.727285 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:36.727399 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:36.729068 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.729801 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.731286 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.731540 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.731701 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:36.736201 (+ 4500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56925: BlockingRecv error: recv got EOF from 127.0.98.2:56925 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 00:47:36.737736 9435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:36.724832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.2:56925)
0927 00:47:36.725409 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:36.725461 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:36.726948 (+ 1487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:36.727619 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:36.727629 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:36.728118 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:36.728892 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:36.728910 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.729978 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.729981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:36.731142 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:36.731156 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.731771 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.731775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:36.731992 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:36.732765 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:36.732784 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:36.736037 (+ 3253us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":163,"threads_started":1}
W20260927 00:47:36.738432 9433 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34583 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34583: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 00:47:36 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:36 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:36.746047 9435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:36.739037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.2:48537)
0927 00:47:36.739244 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:36.739267 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:36.739404 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:36.739724 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:36.739731 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:36.739982 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:36.740162 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:36.740173 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.741134 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.741137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:36.742106 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:36.742117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.742672 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.742676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:36.742806 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:36.743406 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:36.743422 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:36.745865 (+ 2443us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91}
W20260927 00:47:36.746377 9302 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:36.739127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48537 (local address 127.0.98.62:34583)
0927 00:47:36.739266 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:36.739270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:36.739350 (+ 80us) server_negotiation.cc:407] Connection header received
0927 00:47:36.739527 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:36.739530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:36.739592 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:36.739668 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:36.740379 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.741008 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.742286 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.742540 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.742640 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:36.746047 (+ 3407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48537: BlockingRecv error: recv got EOF from 127.0.98.2:48537 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 00:47:36 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:36 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:36.752538 9435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:36.746833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.2:56075)
0927 00:47:36.747068 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:36.747082 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:36.747196 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:36.747457 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:36.747460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:36.747660 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:36.747760 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:36.747766 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.748616 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.748620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:36.749298 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:36.749308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.749749 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.749753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:36.749837 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:36.750389 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:36.750409 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:36.752381 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 00:47:36.752811 9433 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:47:36.752856 9302 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:36.746927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56075 (local address 127.0.98.62:34583)
0927 00:47:36.747048 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:36.747052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:36.747140 (+ 88us) server_negotiation.cc:407] Connection header received
0927 00:47:36.747301 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:36.747304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:36.747356 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:36.747441 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:36.747870 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.748506 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.749414 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.749642 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.749747 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:36.752565 (+ 2818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56075: BlockingRecv error: recv got EOF from 127.0.98.2:56075 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 00:47:36.875536 9436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:36.875962 9436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:36.876016 9436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:36.880630 9436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:36.880743 9436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:36.880816 9436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:47:36.887356 9436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37143
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38291
--tserver_master_addrs=127.0.98.62:34583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:36.888981 9436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:36.890472 9436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:36.903281 9441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:36.903273 9442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:36.903424 9436 server_base.cc:1034] running on GCE node
W20260927 00:47:36.903734 9444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:36.904378 9436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:36.905366 9436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:36.906563 9436 hybrid_clock.cc:648] HybridClock initialized: now 1790470056906539 us; error 54 us; skew 500 ppm
Sep 27 00:47:36 dist-test-slave-86rs krb5kdc[8569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470056, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:36.910228 9436 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 00:47:36.911451 9436 webserver.cc:460] Webserver started at http://127.0.98.3:38291/ using document root <none> and password file <none>
I20260927 00:47:36.912161 9436 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:36.912297 9436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:36.917999 9436 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260927 00:47:36.921036 9451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:36.922868 9436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:47:36.923080 9436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "cb1e88594ee14a7097ee9a8cac788f41"
format_stamp: "Formatted at 2026-09-27 00:47:35 on dist-test-slave-86rs"
server_key: "edd0d0e9ec3c5f5682f6b24b08b54e9d"
server_key_iv: "8aee9bf3b3a71718e9d37b2ca7364813"
server_key_version: "encryptionkey@0"
I20260927 00:47:36.923844 9436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:36.938994 9436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:36.943552 9436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:36.943837 9436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:36.944797 9436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:36.945968 9436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:36.946028 9436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:36.946115 9436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:36.946157 9436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:36.958297 9436 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37143
I20260927 00:47:36.958321 9564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37143 every 8 connection(s)
I20260927 00:47:36.959761 9436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:47:36.967537 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 9436
Sep 27 00:47:36 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:36 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 00:47:36.975199 9302 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:36.962295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45929 (local address 127.0.98.62:34583)
0927 00:47:36.962464 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:36.962468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:36.963566 (+ 1098us) server_negotiation.cc:407] Connection header received
0927 00:47:36.965252 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:36.965257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:36.965340 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:36.965431 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:36.967103 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.967688 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.969520 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.969826 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.969908 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:36.975048 (+ 5140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45929: BlockingRecv error: recv got EOF from 127.0.98.3:45929 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:47:36.976272 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:36.962775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.3:45929)
0927 00:47:36.963349 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:36.963403 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:36.964940 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:36.965673 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:36.965683 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:36.966156 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:36.966914 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:36.966936 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.967902 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.967949 (+ 47us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:36.969410 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:36.969422 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.970067 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.970073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:36.970339 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:36.971168 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:36.971187 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:36.974913 (+ 3726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":143,"threads_started":1}
W20260927 00:47:36.976994 9565 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34583 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34583: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260927 00:47:36.980453 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:36.969115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50752 (local address 127.0.98.62:34583)
0927 00:47:36.969386 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:36.969389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:36.969401 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:47:36.969435 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:36.969438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:36.969502 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:36.969614 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:36.970706 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.971401 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.972859 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.973240 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.973350 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:36.974829 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:36.974836 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:36.974858 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:36.974894 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:36.977128 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:36.977849 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:36.977863 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:36.977870 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:36.978135 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:36.978572 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:36.978575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:36.978576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:36.979326 (+ 750us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:36.979585 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:36.979921 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 00:47:36.980283 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":197,"thread_start_us":117,"threads_started":1}
Sep 27 00:47:36 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:36 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:36.984344 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:36.977655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.3:37059)
0927 00:47:36.977916 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:36.977933 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:36.978023 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:36.978297 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:36.978302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:36.978552 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:36.978700 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:36.978707 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.979832 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.979837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:36.980701 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:36.980714 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.981238 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.981241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:36.981325 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:36.981914 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:36.981927 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:36.984105 (+ 2178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168}
W20260927 00:47:36.984406 9302 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:36.977898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37059 (local address 127.0.98.62:34583)
0927 00:47:36.978045 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:36.978050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:36.978064 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:47:36.978101 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:36.978104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:36.978150 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:36.978234 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:36.978833 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.979683 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.980823 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.981127 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.981233 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:36.984299 (+ 3066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37059: BlockingRecv error: recv got EOF from 127.0.98.3:37059 (error 108)
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":55}
Sep 27 00:47:36 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:36 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:36.991034 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:36.985117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.3:58041)
0927 00:47:36.985265 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:36.985291 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:36.985403 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:36.985655 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:36.985658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:36.985837 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:36.985926 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:36.985932 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.986710 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.986714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:36.987282 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:36.987290 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.987757 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.987760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:36.987837 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:36.988506 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:36.988519 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:36.990828 (+ 2309us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
W20260927 00:47:36.991122 9302 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:36.985229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58041 (local address 127.0.98.62:34583)
0927 00:47:36.985388 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:36.985393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:36.985407 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:47:36.985494 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:36.985497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:36.985548 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:36.985638 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:36.986037 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.986593 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.987408 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:36.987644 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:36.987753 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:36.990986 (+ 3233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58041: BlockingRecv error: recv got EOF from 127.0.98.3:58041 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260927 00:47:36.991389 9565 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 00:47:37 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:37 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:37.522190 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:37.515275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.1:47305)
0927 00:47:37.515653 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:37.515671 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:37.515806 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:37.516316 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:37.516321 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:37.516576 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:37.516746 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:37.516753 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:37.517875 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:37.517880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:37.518609 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:37.518623 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:37.519164 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:37.519167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:37.519250 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:37.519836 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:37.519848 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:37.522033 (+ 2185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":139,"threads_started":1}
W20260927 00:47:37.522357 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:37.515404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47305 (local address 127.0.98.62:34583)
0927 00:47:37.515742 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:37.515746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:37.515765 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:47:37.515984 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:37.515987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:37.516054 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:37.516161 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:37.516895 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:37.517751 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:37.518747 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:37.519050 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:37.519145 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:37.522203 (+ 3058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47305: BlockingRecv error: recv got EOF from 127.0.98.1:47305 (error 108)
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":162,"threads_started":1}
Sep 27 00:47:37 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:37 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:37.760520 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:37.753671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.2:56901)
0927 00:47:37.754011 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:37.754026 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:37.754204 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:37.754595 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:37.754598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:37.754841 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:37.754996 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:37.755010 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:37.755936 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:37.755940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:37.756667 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:37.756678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:37.757253 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:37.757256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:37.757346 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:37.758066 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:37.758082 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:37.760332 (+ 2250us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":129,"threads_started":1}
W20260927 00:47:37.760699 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:37.753760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56901 (local address 127.0.98.62:34583)
0927 00:47:37.753952 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:37.753958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:37.754166 (+ 208us) server_negotiation.cc:407] Connection header received
0927 00:47:37.754359 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:37.754362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:37.754421 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:37.754508 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:37.755181 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:37.755792 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:37.756823 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:37.757120 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:37.757226 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:37.760517 (+ 3291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56901: BlockingRecv error: recv got EOF from 127.0.98.2:56901 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 00:47:37 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:37 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:37.998973 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:37.992331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.3:40337)
0927 00:47:37.992678 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:37.992700 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:37.992838 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:37.993185 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:37.993188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:37.993433 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:37.993588 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:37.993595 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:37.994545 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:37.994548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:37.995163 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:37.995173 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:37.995635 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:37.995638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:37.995722 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:37.996448 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:37.996461 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:37.998764 (+ 2303us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":137,"threads_started":1}
W20260927 00:47:37.999137 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:37.992429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40337 (local address 127.0.98.62:34583)
0927 00:47:37.992589 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:37.992594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:37.992856 (+ 262us) server_negotiation.cc:407] Connection header received
0927 00:47:37.992919 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:37.992921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:37.993037 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:37.993124 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:37.993760 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:37.994407 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:37.995277 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:37.995520 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:37.995607 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:37.998953 (+ 3346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40337: BlockingRecv error: recv got EOF from 127.0.98.3:40337 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 00:47:38 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:38 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:38.534301 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:38.523232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.1:46781)
0927 00:47:38.523591 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:38.523607 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:38.523741 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:38.524115 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:38.524121 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:38.524452 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:38.524635 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:38.524646 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.525531 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.525534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:38.526143 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:38.526151 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.526691 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.526694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:38.526771 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:38.531819 (+ 5048us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:38.531837 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:38.534108 (+ 2271us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":119,"threads_started":1}
W20260927 00:47:38.534498 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:38.523301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46781 (local address 127.0.98.62:34583)
0927 00:47:38.523650 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:38.523655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:38.523673 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:47:38.523856 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:38.523860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:38.523936 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:38.524063 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:38.524785 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.525406 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.526263 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.526552 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.526650 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:38.534339 (+ 7689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46781: BlockingRecv error: recv got EOF from 127.0.98.1:46781 (error 108)
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":130,"threads_started":1}
Sep 27 00:47:38 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:38 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:38.768286 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:38.761580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.2:58539)
0927 00:47:38.761978 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:38.761994 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:38.762170 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:38.762488 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:38.762492 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:38.762758 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:38.762896 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:38.762906 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.763770 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.763776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:38.764492 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:38.764502 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.765005 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.765009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:38.765107 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:38.765804 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:38.765825 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:38.768019 (+ 2194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":111,"threads_started":1}
W20260927 00:47:38.768339 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:38.761669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58539 (local address 127.0.98.62:34583)
0927 00:47:38.761865 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:38.761872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:38.762090 (+ 218us) server_negotiation.cc:407] Connection header received
0927 00:47:38.762286 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:38.762291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:38.762355 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:38.762458 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:38.763031 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.763633 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.764617 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:38.764866 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:38.764995 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:38.768181 (+ 3186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58539: BlockingRecv error: recv got EOF from 127.0.98.2:58539 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 27 00:47:39 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:39 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:39.007956 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:39.000332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.3:47061)
0927 00:47:39.000756 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:39.000780 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:39.000957 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:39.001268 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:39.001271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:39.001564 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:39.001682 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:39.001689 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:39.002703 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:39.002709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:39.003669 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:39.003682 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:39.004185 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:39.004188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:39.004339 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:39.005063 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:39.005076 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:39.007712 (+ 2636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":139,"threads_started":1}
W20260927 00:47:39.008057 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:39.000624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47061 (local address 127.0.98.62:34583)
0927 00:47:39.000782 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:39.000788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:39.000876 (+ 88us) server_negotiation.cc:407] Connection header received
0927 00:47:39.001096 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:39.001099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:39.001154 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:39.001241 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:39.001805 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:39.002570 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:39.003801 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:39.004065 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:39.004152 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:39.007933 (+ 3781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47061: BlockingRecv error: recv got EOF from 127.0.98.3:47061 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 00:47:39 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:39 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:39.541848 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:39.535440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.1:34055)
0927 00:47:39.535836 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:39.535857 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:39.536028 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:39.536439 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:39.536442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:39.536684 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:39.536813 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:39.536819 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:39.537661 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:39.537665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:39.538309 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:39.538319 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:39.538804 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:39.538807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:39.538880 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:39.539476 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:39.539490 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:39.541677 (+ 2187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":109,"threads_started":1}
W20260927 00:47:39.542052 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:39.535476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34055 (local address 127.0.98.62:34583)
0927 00:47:39.535801 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:39.535807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:39.535966 (+ 159us) server_negotiation.cc:407] Connection header received
0927 00:47:39.536144 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:39.536147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:39.536216 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:39.536376 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:39.536914 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:39.537531 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:39.538422 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:39.538686 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:39.538776 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:39.541863 (+ 3087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34055: BlockingRecv error: recv got EOF from 127.0.98.1:34055 (error 108)
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":97,"threads_started":1}
Sep 27 00:47:39 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:39 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:39.779793 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:39.772733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.2:41543)
0927 00:47:39.773334 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:39.773352 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:39.773491 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:39.773999 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:39.774002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:39.774322 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:39.774481 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:39.774489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:39.775378 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:39.775381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:39.776067 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:39.776077 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:39.776626 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:39.776630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:39.776714 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:39.777376 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:39.777390 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:39.779638 (+ 2248us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":126,"threads_started":1}
W20260927 00:47:39.780040 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:39.773210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41543 (local address 127.0.98.62:34583)
0927 00:47:39.773722 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:39.773727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:39.773743 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:47:39.773787 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:39.773790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:39.773849 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:39.773927 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:39.774625 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:39.775233 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:39.776208 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:39.776499 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:39.776592 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:39.779872 (+ 3280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41543: BlockingRecv error: recv got EOF from 127.0.98.2:41543 (error 108)
Metrics: {"server-negotiator.queue_time_us":413}
Sep 27 00:47:40 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:40 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:40.017721 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:40.009114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.3:58251)
0927 00:47:40.009556 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:40.009572 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:40.009736 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:40.010296 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:40.010302 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:40.010623 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:40.010840 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:40.010853 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:40.012039 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:40.012047 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:40.013300 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:40.013319 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:40.013887 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:40.013893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:40.014040 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:40.014735 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:40.014759 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:40.017491 (+ 2732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":110,"threads_started":1}
W20260927 00:47:40.017786 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:40.009231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58251 (local address 127.0.98.62:34583)
0927 00:47:40.009430 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:40.009438 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:40.009660 (+ 222us) server_negotiation.cc:407] Connection header received
0927 00:47:40.009865 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:40.009871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:40.009970 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:40.010096 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:40.011029 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:40.011893 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:40.013486 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:40.013737 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:40.013838 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:40.017650 (+ 3812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58251: BlockingRecv error: recv got EOF from 127.0.98.3:58251 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 00:47:40 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:40 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:40.548774 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:40.542834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.1:58811)
0927 00:47:40.543151 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:40.543166 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:40.543283 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:40.543535 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:40.543537 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:40.543763 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:40.543887 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:40.543893 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:40.544731 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:40.544735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:40.545332 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:40.545352 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:40.545809 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:40.545812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:40.545881 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:40.546462 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:40.546474 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:40.548609 (+ 2135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":148,"threads_started":1}
W20260927 00:47:40.548921 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:40.542867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58811 (local address 127.0.98.62:34583)
0927 00:47:40.543167 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:40.543172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:40.543238 (+ 66us) server_negotiation.cc:407] Connection header received
0927 00:47:40.543367 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:40.543369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:40.543422 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:40.543523 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:40.543989 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:40.544612 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:40.545459 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:40.545712 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:40.545794 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:40.548774 (+ 2980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58811: BlockingRecv error: recv got EOF from 127.0.98.1:58811 (error 108)
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":92,"threads_started":1}
Sep 27 00:47:40 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:40 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:40.787635 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:40.781004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.2:41359)
0927 00:47:40.781395 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:40.781415 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:40.781552 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:40.781814 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:40.781816 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:40.782045 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:40.782171 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:40.782176 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:40.783028 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:40.783034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:40.783804 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:40.783817 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:40.784352 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:40.784355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:40.784435 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:40.785090 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:40.785104 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:40.787459 (+ 2355us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":161,"threads_started":1}
W20260927 00:47:40.787806 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:40.781085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41359 (local address 127.0.98.62:34583)
0927 00:47:40.781243 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:40.781248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:40.781508 (+ 260us) server_negotiation.cc:407] Connection header received
0927 00:47:40.781641 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:40.781643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:40.781696 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:40.781777 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:40.782284 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:40.782883 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:40.783943 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:40.784201 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:40.784343 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:40.787643 (+ 3300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41359: BlockingRecv error: recv got EOF from 127.0.98.2:41359 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 00:47:41 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:41 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:41.025869 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:41.019012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.3:53427)
0927 00:47:41.019340 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:41.019356 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:41.019527 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:41.019872 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:41.019876 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:41.020210 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:41.020461 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:41.020472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:41.021395 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:41.021399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:41.022113 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:41.022123 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:41.022693 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:41.022697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:41.022777 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:41.023433 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:41.023446 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:41.025698 (+ 2252us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":117,"threads_started":1}
W20260927 00:47:41.026048 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:41.019126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53427 (local address 127.0.98.62:34583)
0927 00:47:41.019295 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:41.019301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:41.019461 (+ 160us) server_negotiation.cc:407] Connection header received
0927 00:47:41.019643 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:41.019646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:41.019721 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:41.019813 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:41.020640 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:41.021261 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:41.022255 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:41.022547 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:41.022654 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:41.025876 (+ 3222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53427: BlockingRecv error: recv got EOF from 127.0.98.3:53427 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 00:47:41 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:41 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:41.556780 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:41.549894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.1:57657)
0927 00:47:41.550222 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:41.550244 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:41.550384 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:41.550772 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:41.550776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:41.551051 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:41.551177 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:41.551184 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:41.552106 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:41.552111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:41.552955 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:41.552966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:41.553566 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:41.553570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:41.553662 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:41.554367 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:41.554381 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:41.556600 (+ 2219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":108,"threads_started":1}
W20260927 00:47:41.557009 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:41.549962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57657 (local address 127.0.98.62:34583)
0927 00:47:41.550336 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:41.550342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:41.550358 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:47:41.550538 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:41.550544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:41.550614 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:41.550741 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:41.551314 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:41.551980 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:41.553095 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:41.553385 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:41.553595 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:41.556798 (+ 3203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57657: BlockingRecv error: recv got EOF from 127.0.98.1:57657 (error 108)
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":112,"threads_started":1}
Sep 27 00:47:41 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:41 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:41.795677 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:41.788679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.2:58391)
0927 00:47:41.789062 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:41.789078 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:41.789206 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:41.789544 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:41.789547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:41.789787 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:41.789917 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:41.789924 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:41.790894 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:41.790898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:41.791611 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:41.791623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:41.792173 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:41.792176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:41.792321 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:41.793040 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:41.793060 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:41.795486 (+ 2426us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":128,"threads_started":1}
W20260927 00:47:41.795902 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:41.788833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58391 (local address 127.0.98.62:34583)
0927 00:47:41.788998 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:41.789003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:41.789159 (+ 156us) server_negotiation.cc:407] Connection header received
0927 00:47:41.789315 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:41.789318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:41.789373 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:41.789442 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:41.790090 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:41.790765 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:41.791758 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:41.792029 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:41.792128 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:41.795705 (+ 3577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58391: BlockingRecv error: recv got EOF from 127.0.98.2:58391 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 00:47:42 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:42 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:42.034366 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:42.027116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.3:46525)
0927 00:47:42.027505 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:42.027521 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:42.027650 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:42.028069 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:42.028071 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:42.028338 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:42.028523 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:42.028536 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:42.029689 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:42.029692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:42.030512 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:42.030526 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:42.031146 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:42.031149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:42.031219 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:42.031818 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:42.031829 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:42.034169 (+ 2340us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":148,"threads_started":1}
W20260927 00:47:42.034485 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:42.027165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46525 (local address 127.0.98.62:34583)
0927 00:47:42.027367 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:42.027372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:42.027613 (+ 241us) server_negotiation.cc:407] Connection header received
0927 00:47:42.027808 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:42.027813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:42.027885 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:42.028041 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:42.028707 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:42.029566 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:42.030614 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:42.030993 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:42.031098 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:42.034354 (+ 3256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46525: BlockingRecv error: recv got EOF from 127.0.98.3:46525 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 00:47:42 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:42 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:42.564447 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:42.557825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.1:51941)
0927 00:47:42.558288 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:42.558322 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:42.558474 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:42.558859 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:42.558864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:42.559164 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:42.559311 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:42.559318 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:42.560154 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:42.560157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:42.560814 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:42.560823 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:42.561389 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:42.561392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:42.561463 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:42.562112 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:42.562126 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:42.564294 (+ 2168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":105,"threads_started":1}
W20260927 00:47:42.564610 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:42.557891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51941 (local address 127.0.98.62:34583)
0927 00:47:42.558253 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:42.558259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:42.558405 (+ 146us) server_negotiation.cc:407] Connection header received
0927 00:47:42.558625 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:42.558628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:42.558701 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:42.558804 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:42.559417 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:42.560030 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:42.560998 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:42.561263 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:42.561375 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:42.564447 (+ 3072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51941: BlockingRecv error: recv got EOF from 127.0.98.1:51941 (error 108)
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":133,"threads_started":1}
Sep 27 00:47:42 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:42 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:42.803233 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:42.796779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.2:44817)
0927 00:47:42.797109 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:42.797123 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:42.797246 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:42.797529 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:42.797531 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:42.797787 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:42.797905 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:42.797912 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:42.798729 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:42.798732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:42.799338 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:42.799347 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:42.799828 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:42.799831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:42.799904 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:42.800577 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:42.800590 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:42.803068 (+ 2478us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":120,"threads_started":1}
W20260927 00:47:42.803570 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:42.796889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44817 (local address 127.0.98.62:34583)
0927 00:47:42.797068 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:42.797072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:42.797202 (+ 130us) server_negotiation.cc:407] Connection header received
0927 00:47:42.797333 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:42.797336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:42.797388 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:42.797479 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:42.798027 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:42.798624 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:42.799469 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:42.799707 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:42.799799 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:42.803287 (+ 3488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44817: BlockingRecv error: recv got EOF from 127.0.98.2:44817 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 00:47:43 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:43 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:43.042622 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:43.035654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.3:45693)
0927 00:47:43.035978 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:43.035993 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:43.036162 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:43.036598 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:43.036601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:43.036880 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:43.037038 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:43.037046 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:43.037874 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:43.037878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:43.038545 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:43.038556 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:43.039062 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:43.039067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:43.039158 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:43.039772 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:43.039785 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:43.042461 (+ 2676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":100,"threads_started":1}
W20260927 00:47:43.043018 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:43.035731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45693 (local address 127.0.98.62:34583)
0927 00:47:43.035913 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:43.035917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:43.036098 (+ 181us) server_negotiation.cc:407] Connection header received
0927 00:47:43.036326 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:43.036330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:43.036417 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:43.036504 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:43.037149 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:43.037742 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:43.038678 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:43.038927 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:43.039017 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:43.042688 (+ 3671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45693: BlockingRecv error: recv got EOF from 127.0.98.3:45693 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 27 00:47:43 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:43 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:43.577661 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:43.565450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.1:41691)
0927 00:47:43.565839 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:43.565855 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:43.566003 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:43.566273 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:43.566275 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:43.566509 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:43.566632 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:43.566638 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:43.567522 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:43.567525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:43.568146 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:43.568156 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:43.568707 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:43.568710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:43.568782 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:43.574847 (+ 6065us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:43.574868 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:43.577506 (+ 2638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":111,"threads_started":1}
W20260927 00:47:43.577913 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:43.565528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41691 (local address 127.0.98.62:34583)
0927 00:47:43.565860 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:43.565865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:43.565933 (+ 68us) server_negotiation.cc:407] Connection header received
0927 00:47:43.566093 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:43.566095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:43.566148 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:43.566235 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:43.566787 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:43.567394 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:43.568291 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:43.568577 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:43.568676 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:43.577716 (+ 9040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41691: BlockingRecv error: recv got EOF from 127.0.98.1:41691 (error 108)
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":101,"threads_started":1}
Sep 27 00:47:43 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:43 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:43.812817 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:43.804255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.2:48481)
0927 00:47:43.804607 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:43.804622 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:43.804802 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:43.805206 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:43.805210 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:43.805507 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:43.805684 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:43.805695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:43.806970 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:43.806974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:43.807975 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:43.807986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:43.808742 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:43.808746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:43.808843 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:43.809611 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:43.809627 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:43.812563 (+ 2936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":97,"threads_started":1}
W20260927 00:47:43.812987 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:43.804332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48481 (local address 127.0.98.62:34583)
0927 00:47:43.804544 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:43.804551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:43.804707 (+ 156us) server_negotiation.cc:407] Connection header received
0927 00:47:43.804926 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:43.804932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:43.805016 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:43.805150 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:43.805835 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:43.806829 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:43.808102 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:43.808581 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:43.808715 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:43.812810 (+ 4095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48481: BlockingRecv error: recv got EOF from 127.0.98.2:48481 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 27 00:47:44 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:44 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:44.051359 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:44.043967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.3:40161)
0927 00:47:44.044569 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:44.044584 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:44.044777 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:44.045170 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:44.045173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:44.045443 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:44.045574 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:44.045580 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:44.046458 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:44.046464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:44.047094 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:44.047105 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:44.047628 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:44.047632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:44.047718 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:44.048544 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:44.048559 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:44.051160 (+ 2601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":151,"threads_started":1}
W20260927 00:47:44.051589 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:44.044197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40161 (local address 127.0.98.62:34583)
0927 00:47:44.044475 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:44.044481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:44.044703 (+ 222us) server_negotiation.cc:407] Connection header received
0927 00:47:44.044908 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:44.044916 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:44.044979 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:44.045135 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:44.045716 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:44.046331 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:44.047238 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:44.047499 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:44.047619 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:44.051437 (+ 3818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40161: BlockingRecv error: recv got EOF from 127.0.98.3:40161 (error 108)
Metrics: {"server-negotiator.queue_time_us":144}
Sep 27 00:47:44 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:44 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:44.585053 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:44.578651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.1:35683)
0927 00:47:44.579060 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:44.579077 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:44.579190 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:44.579486 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:44.579489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:44.579718 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:44.579840 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:44.579846 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:44.580814 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:44.580817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:44.581434 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:44.581443 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:44.581899 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:44.581902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:44.581969 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:44.582674 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:44.582687 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:44.584900 (+ 2213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":155,"threads_started":1}
W20260927 00:47:44.585282 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:44.578824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35683 (local address 127.0.98.62:34583)
0927 00:47:44.579111 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:44.579115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:44.579131 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:47:44.579295 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:44.579297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:44.579347 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:44.579435 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:44.579971 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:44.580703 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:44.581545 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:44.581795 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:44.581880 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:44.585081 (+ 3201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35683: BlockingRecv error: recv got EOF from 127.0.98.1:35683 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":71,"threads_started":1}
Sep 27 00:47:44 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:44 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:44.820927 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:44.813985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.2:52079)
0927 00:47:44.814374 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:44.814391 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:44.814536 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:44.814800 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:44.814808 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:44.815098 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:44.815223 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:44.815230 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:44.816295 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:44.816299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:44.817005 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:44.817016 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:44.817541 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:44.817545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:44.817623 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:44.818231 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:44.818243 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:44.820747 (+ 2504us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":93,"threads_started":1}
W20260927 00:47:44.821147 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:44.814191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52079 (local address 127.0.98.62:34583)
0927 00:47:44.814359 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:44.814363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:44.814474 (+ 111us) server_negotiation.cc:407] Connection header received
0927 00:47:44.814628 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:44.814631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:44.814685 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:44.814756 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:44.815373 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:44.816134 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:44.817127 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:44.817383 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:44.817477 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:44.821021 (+ 3544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52079: BlockingRecv error: recv got EOF from 127.0.98.2:52079 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 00:47:45 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:45 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:45.059588 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:45.052632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.3:33527)
0927 00:47:45.053075 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:45.053093 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:45.053267 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:45.053609 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:45.053613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:45.053877 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:45.053990 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:45.053997 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:45.054832 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:45.054837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:45.055496 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:45.055504 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:45.055988 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:45.055992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:45.056096 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:45.056805 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:45.056819 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:45.059416 (+ 2597us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":165,"threads_started":1}
W20260927 00:47:45.059753 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:45.052778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33527 (local address 127.0.98.62:34583)
0927 00:47:45.052984 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:45.052990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:45.053198 (+ 208us) server_negotiation.cc:407] Connection header received
0927 00:47:45.053407 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:45.053409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:45.053466 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:45.053583 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:45.054124 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:45.054715 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:45.055627 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:45.055869 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:45.055982 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:45.059606 (+ 3624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33527: BlockingRecv error: recv got EOF from 127.0.98.3:33527 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 27 00:47:45 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:45 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:45.592911 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:45.586147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.1:37485)
0927 00:47:45.586526 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:45.586544 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:45.586676 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:45.587044 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:45.587046 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:45.587270 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:45.587396 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:45.587402 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:45.588395 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:45.588398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:45.589017 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:45.589027 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:45.589592 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:45.589595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:45.589666 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:45.590290 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:45.590303 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:45.592763 (+ 2460us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":104,"threads_started":1}
W20260927 00:47:45.593096 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:45.586227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37485 (local address 127.0.98.62:34583)
0927 00:47:45.586594 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:45.586599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:45.586619 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:47:45.586786 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:45.586788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:45.586844 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:45.586986 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:45.587533 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:45.588257 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:45.589176 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:45.589478 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:45.589573 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:45.592922 (+ 3349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37485: BlockingRecv error: recv got EOF from 127.0.98.1:37485 (error 108)
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":126,"threads_started":1}
Sep 27 00:47:45 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:45 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:45.829247 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:45.822122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.2:53499)
0927 00:47:45.822514 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:45.822530 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:45.822661 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:45.823007 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:45.823010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:45.823245 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:45.823367 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:45.823373 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:45.824301 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:45.824313 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:45.824982 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:45.824992 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:45.825517 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:45.825520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:45.825600 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:45.826271 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:45.826285 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:45.829063 (+ 2778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":143,"threads_started":1}
W20260927 00:47:45.829478 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:45.822212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53499 (local address 127.0.98.62:34583)
0927 00:47:45.822394 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:45.822398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:45.822612 (+ 214us) server_negotiation.cc:407] Connection header received
0927 00:47:45.822804 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:45.822807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:45.822865 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:45.822986 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:45.823485 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:45.824114 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:45.825125 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:45.825381 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:45.825472 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:45.829280 (+ 3808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53499: BlockingRecv error: recv got EOF from 127.0.98.2:53499 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 00:47:46 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:46 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:46.068285 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:46.060799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.3:40845)
0927 00:47:46.061453 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:46.061468 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:46.061655 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:46.061978 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:46.061982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:46.062248 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:46.062371 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:46.062378 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:46.063352 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:46.063357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:46.063976 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:46.063986 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:46.064609 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:46.064613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:46.064701 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:46.065372 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:46.065384 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:46.068044 (+ 2660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":337,"threads_started":1}
W20260927 00:47:46.068321 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:46.061119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40845 (local address 127.0.98.62:34583)
0927 00:47:46.061327 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:46.061333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:46.061578 (+ 245us) server_negotiation.cc:407] Connection header received
0927 00:47:46.061779 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:46.061785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:46.061850 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:46.061952 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:46.062511 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:46.063232 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:46.064120 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:46.064464 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:46.064599 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:46.068181 (+ 3582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40845: BlockingRecv error: recv got EOF from 127.0.98.3:40845 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 00:47:46 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:46 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:46.601303 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:46.594085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.1:58745)
0927 00:47:46.594419 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:46.594434 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:46.594575 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:46.594898 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:46.594901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:46.595134 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:46.595262 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:46.595269 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:46.596167 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:46.596170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:46.596892 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:46.596904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:46.597505 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:46.597509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:46.597591 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:46.598337 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:46.598351 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:46.601138 (+ 2787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":110,"threads_started":1}
W20260927 00:47:46.601567 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:46.594127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58745 (local address 127.0.98.62:34583)
0927 00:47:46.594403 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:46.594407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:46.594531 (+ 124us) server_negotiation.cc:407] Connection header received
0927 00:47:46.594664 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:46.594666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:46.594728 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:46.594821 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:46.595409 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:46.596040 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:46.597051 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:46.597333 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:46.597440 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:46.601365 (+ 3925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58745: BlockingRecv error: recv got EOF from 127.0.98.1:58745 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":110,"threads_started":1}
Sep 27 00:47:46 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:46 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:46.836803 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:46.830398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.2:46967)
0927 00:47:46.830769 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:46.830791 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:46.830950 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:46.831253 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:46.831256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:46.831511 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:46.831632 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:46.831638 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:46.832563 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:46.832567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:46.833184 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:46.833193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:46.833669 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:46.833673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:46.833768 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:46.834442 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:46.834456 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:46.836650 (+ 2194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":110,"threads_started":1}
W20260927 00:47:46.836988 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:46.830493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46967 (local address 127.0.98.62:34583)
0927 00:47:46.830723 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:46.830728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:46.830878 (+ 150us) server_negotiation.cc:407] Connection header received
0927 00:47:46.831067 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:46.831074 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:46.831130 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:46.831227 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:46.831753 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:46.832443 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:46.833306 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:46.833551 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:46.833662 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:46.836830 (+ 3168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46967: BlockingRecv error: recv got EOF from 127.0.98.2:46967 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 27 00:47:47 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:47 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:47.076938 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:47.069451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.3:58727)
0927 00:47:47.069987 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:47.070008 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:47.070153 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:47.070701 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:47.070704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:47.071008 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:47.071163 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:47.071172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:47.072329 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:47.072333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:47.072959 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:47.072968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:47.073484 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:47.073488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:47.073570 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:47.074304 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:47.074329 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:47.076777 (+ 2448us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":100,"threads_started":1}
W20260927 00:47:47.077157 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:47.069748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58727 (local address 127.0.98.62:34583)
0927 00:47:47.070393 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:47.070399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:47.070417 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:47:47.070477 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:47.070480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:47.070544 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:47.070640 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:47.071329 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:47.072164 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:47.073092 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:47.073333 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:47.073424 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:47.076990 (+ 3566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58727: BlockingRecv error: recv got EOF from 127.0.98.3:58727 (error 108)
Metrics: {"server-negotiator.queue_time_us":480}
Sep 27 00:47:47 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:47 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:47.609531 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:47.602428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.1:59387)
0927 00:47:47.602769 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:47.602783 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:47.602901 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:47.603397 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:47.603403 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:47.603675 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:47.603802 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:47.603808 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:47.604803 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:47.604806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:47.605429 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:47.605465 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:47.606094 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:47.606097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:47.606168 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:47.606857 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:47.606869 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:47.609311 (+ 2442us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":126,"threads_started":1}
W20260927 00:47:47.609599 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:47.602549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59387 (local address 127.0.98.62:34583)
0927 00:47:47.602827 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:47.602831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:47.602850 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:47:47.603118 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:47.603120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:47.603181 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:47.603333 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:47.603970 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:47.604684 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:47.605610 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:47.605920 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:47.606093 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:47.609460 (+ 3367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59387: BlockingRecv error: recv got EOF from 127.0.98.1:59387 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":97,"threads_started":1}
Sep 27 00:47:47 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:47 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:47.844694 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:47.837958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.2:38631)
0927 00:47:47.838312 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:47.838326 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:47.838469 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:47.838828 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:47.838831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:47.839072 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:47.839196 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:47.839202 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:47.840063 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:47.840066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:47.840833 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:47.840844 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:47.841432 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:47.841435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:47.841529 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:47.842184 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:47.842199 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:47.844538 (+ 2339us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":128,"threads_started":1}
W20260927 00:47:47.844902 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:47.838074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38631 (local address 127.0.98.62:34583)
0927 00:47:47.838273 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:47.838277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:47.838407 (+ 130us) server_negotiation.cc:407] Connection header received
0927 00:47:47.838584 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:47.838586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:47.838659 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:47.838741 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:47.839320 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:47.839928 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:47.840986 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:47.841257 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:47.841359 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:47.844771 (+ 3412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38631: BlockingRecv error: recv got EOF from 127.0.98.2:38631 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 27 00:47:48 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:48 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:48.085664 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:48.078025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.3:50487)
0927 00:47:48.078601 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:48.078616 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:48.078775 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:48.079142 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:48.079146 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:48.079471 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:48.079623 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:48.079632 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:48.080677 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:48.080681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:48.081371 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:48.081382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:48.081960 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:48.081965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:48.082079 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:48.082809 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:48.082823 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:48.085394 (+ 2571us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":155,"threads_started":1}
W20260927 00:47:48.085745 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:48.078193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50487 (local address 127.0.98.62:34583)
0927 00:47:48.078358 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:48.078365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:48.078705 (+ 340us) server_negotiation.cc:407] Connection header received
0927 00:47:48.078886 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:48.078890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:48.078967 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:48.079056 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:48.079770 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:48.080538 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:48.081504 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:48.081789 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:48.081946 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:48.085607 (+ 3661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50487: BlockingRecv error: recv got EOF from 127.0.98.3:50487 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 00:47:48 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:48 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:48.621587 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:48.610854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.1:49175)
0927 00:47:48.611218 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:48.611235 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:48.611363 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:48.611677 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:48.611679 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:48.611908 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:48.612084 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:48.612092 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:48.613096 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:48.613100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:48.613763 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:48.613776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:48.614364 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:48.614368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:48.614462 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:48.618826 (+ 4364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:48.618843 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:48.621368 (+ 2525us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":115,"threads_started":1}
W20260927 00:47:48.621714 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:48.611018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49175 (local address 127.0.98.62:34583)
0927 00:47:48.611324 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:48.611328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:48.611345 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:47:48.611475 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:48.611478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:48.611532 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:48.611623 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:48.612329 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:48.612955 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:48.613912 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:48.614212 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:48.614307 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:48.621561 (+ 7254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49175: BlockingRecv error: recv got EOF from 127.0.98.1:49175 (error 108)
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":76,"threads_started":1}
Sep 27 00:47:48 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:48 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:48.852169 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:48.845711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.2:57683)
0927 00:47:48.846059 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:48.846077 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:48.846265 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:48.846582 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:48.846585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:48.846833 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:48.846955 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:48.846963 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:48.847777 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:48.847782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:48.848461 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:48.848470 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:48.848943 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:48.848947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:48.849028 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:48.849752 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:48.849765 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:48.851971 (+ 2206us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":106,"threads_started":1}
W20260927 00:47:48.852344 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:48.845796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57683 (local address 127.0.98.62:34583)
0927 00:47:48.846040 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:48.846046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:48.846180 (+ 134us) server_negotiation.cc:407] Connection header received
0927 00:47:48.846381 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:48.846387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:48.846458 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:48.846557 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:48.847087 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:48.847664 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:48.848586 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:48.848826 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:48.848978 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:48.852159 (+ 3181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57683: BlockingRecv error: recv got EOF from 127.0.98.2:57683 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 27 00:47:49 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:49 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:49.094280 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:49.086816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.3:42177)
0927 00:47:49.087180 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:49.087201 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:49.087348 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:49.087671 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:49.087674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:49.087902 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:49.088034 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:49.088040 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:49.088956 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:49.088959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:49.089619 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:49.089630 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:49.090419 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:49.090423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:49.090498 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:49.091201 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:49.091220 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:49.094108 (+ 2888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":96,"threads_started":1}
W20260927 00:47:49.094535 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:49.086984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42177 (local address 127.0.98.62:34583)
0927 00:47:49.087177 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:49.087181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:49.087331 (+ 150us) server_negotiation.cc:407] Connection header received
0927 00:47:49.087477 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:49.087480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:49.087543 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:49.087628 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:49.088152 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:49.088818 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:49.089749 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:49.090004 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:49.090092 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:49.094348 (+ 4256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42177: BlockingRecv error: recv got EOF from 127.0.98.3:42177 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 00:47:49 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:49 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:49.630009 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:49.622683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.1:38301)
0927 00:47:49.623091 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:49.623113 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:49.623250 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:49.623671 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:49.623674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:49.623898 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:49.624060 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:49.624068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:49.625077 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:49.625107 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:49.626158 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:49.626171 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:49.626748 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:49.626750 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:49.626819 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:49.627472 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:49.627490 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:49.629805 (+ 2315us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":141,"threads_started":1}
W20260927 00:47:49.630179 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:49.622798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38301 (local address 127.0.98.62:34583)
0927 00:47:49.623186 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:49.623192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:49.623211 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:47:49.623436 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:49.623440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:49.623499 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:49.623597 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:49.624216 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:49.624935 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:49.626336 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:49.626635 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:49.626729 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:49.629993 (+ 3264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38301: BlockingRecv error: recv got EOF from 127.0.98.1:38301 (error 108)
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":144,"threads_started":1}
Sep 27 00:47:49 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:49 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:49.860915 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:49.853358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.2:42953)
0927 00:47:49.853758 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:49.853777 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:49.853983 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:49.854341 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:49.854344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:49.854650 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:49.854801 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:49.854809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:49.855677 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:49.855680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:49.856595 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:49.856606 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:49.857245 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:49.857248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:49.857330 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:49.858025 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:49.858038 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:49.860752 (+ 2714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":129,"threads_started":1}
W20260927 00:47:49.861157 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:49.853612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42953 (local address 127.0.98.62:34583)
0927 00:47:49.853776 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:49.853781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:49.853894 (+ 113us) server_negotiation.cc:407] Connection header received
0927 00:47:49.854147 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:49.854151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:49.854217 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:49.854313 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:49.854952 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:49.855558 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:49.856742 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:49.857091 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:49.857199 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:49.860991 (+ 3792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42953: BlockingRecv error: recv got EOF from 127.0.98.2:42953 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 00:47:50 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:50 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:50.102527 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:50.095526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.3:39843)
0927 00:47:50.095922 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:50.095945 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:50.096079 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:50.096428 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:50.096430 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:50.096713 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:50.096848 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:50.096856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:50.097843 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:50.097847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:50.098508 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:50.098518 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:50.099028 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:50.099031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:50.099113 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:50.099753 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:50.099765 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:50.102298 (+ 2533us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":167,"threads_started":1}
W20260927 00:47:50.102677 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:50.095620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39843 (local address 127.0.98.62:34583)
0927 00:47:50.095823 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:50.095828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:50.096027 (+ 199us) server_negotiation.cc:407] Connection header received
0927 00:47:50.096173 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:50.096176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:50.096292 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:50.096374 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:50.097038 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:50.097694 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:50.098656 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:50.098905 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:50.099002 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:50.102522 (+ 3520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39843: BlockingRecv error: recv got EOF from 127.0.98.3:39843 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 00:47:50 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:50 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:50.638139 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:50.631181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.1:54097)
0927 00:47:50.631573 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:50.631597 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:50.631732 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:50.632051 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:50.632053 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:50.632374 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:50.632514 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:50.632523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:50.633429 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:50.633433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:50.634137 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:50.634148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:50.634682 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:50.634685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:50.634766 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:50.635413 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:50.635427 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:50.637926 (+ 2499us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":109,"threads_started":1}
W20260927 00:47:50.638309 9623 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:50.631256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54097 (local address 127.0.98.62:34583)
0927 00:47:50.631664 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:50.631668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:50.631683 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:47:50.631825 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:50.631827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:50.631877 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:50.631993 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:50.632651 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:50.633298 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:50.634287 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:50.634544 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:50.634643 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:50.638151 (+ 3508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54097: BlockingRecv error: recv got EOF from 127.0.98.1:54097 (error 108)
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":97,"threads_started":1}
Sep 27 00:47:50 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:50 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:50.868745 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:50.861975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.2:46577)
0927 00:47:50.862397 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:50.862413 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:50.862563 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:50.862856 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:50.862858 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:50.863107 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:50.863229 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:50.863236 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:50.864192 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:50.864195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:50.864893 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:50.864902 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:50.865483 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:50.865486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:50.865565 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:50.866190 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:50.866203 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:50.868588 (+ 2385us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":158,"threads_started":1}
W20260927 00:47:50.868978 9623 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:50.862033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46577 (local address 127.0.98.62:34583)
0927 00:47:50.862230 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:50.862236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:50.862525 (+ 289us) server_negotiation.cc:407] Connection header received
0927 00:47:50.862663 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:50.862666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:50.862725 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:50.862805 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:50.863388 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:50.864055 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:50.865038 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:50.865328 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:50.865437 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:50.868776 (+ 3339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46577: BlockingRecv error: recv got EOF from 127.0.98.2:46577 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 00:47:51 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:51 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:51.111496 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:51.103916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.3:52107)
0927 00:47:51.104738 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:51.104760 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:51.104923 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:51.105267 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:51.105270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:51.105629 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:51.105768 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:51.105775 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:51.107002 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:51.107006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:51.107754 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:51.107765 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:51.108308 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:51.108312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:51.108383 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:51.108979 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:51.109027 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:51.111322 (+ 2295us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":149,"threads_started":1}
W20260927 00:47:51.111789 9623 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:51.104187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52107 (local address 127.0.98.62:34583)
0927 00:47:51.104405 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:51.104411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:51.104868 (+ 457us) server_negotiation.cc:407] Connection header received
0927 00:47:51.105068 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:51.105073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:51.105146 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:51.105242 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:51.105970 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:51.106852 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:51.107874 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:51.108150 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:51.108299 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:51.111543 (+ 3244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52107: BlockingRecv error: recv got EOF from 127.0.98.3:52107 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Sep 27 00:47:51 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:51 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:51.645851 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:51.639335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.1:44559)
0927 00:47:51.639665 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:51.639680 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:51.639785 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:51.640118 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:51.640120 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:51.640378 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:51.640525 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:51.640531 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:51.641435 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:51.641438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:51.642082 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:51.642092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:51.642551 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:51.642554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:51.642626 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:51.643241 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:51.643254 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:51.645621 (+ 2367us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":116,"threads_started":1}
W20260927 00:47:51.645946 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:51.639466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44559 (local address 127.0.98.62:34583)
0927 00:47:51.639736 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:51.639740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:51.639756 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:47:51.639896 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:51.639899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:51.639954 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:51.640057 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:51.640664 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:51.641290 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:51.642196 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:51.642433 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:51.642521 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:51.645789 (+ 3268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44559: BlockingRecv error: recv got EOF from 127.0.98.1:44559 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":105,"threads_started":1}
Sep 27 00:47:51 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:47:51 dist-test-slave-86rs krb5kdc[8569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:47:51.876752 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:51.869980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.2:47115)
0927 00:47:51.870488 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:51.870506 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:51.870640 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:51.870999 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:51.871002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:51.871252 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:51.871371 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:51.871378 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:51.872206 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:51.872208 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:51.872858 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:51.872867 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:51.873396 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:51.873398 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:51.873486 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:51.874197 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:51.874211 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:51.876576 (+ 2365us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34583: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":127,"threads_started":1}
W20260927 00:47:51.877043 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:51.870016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47115 (local address 127.0.98.62:34583)
0927 00:47:51.870204 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:51.870209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:51.870594 (+ 385us) server_negotiation.cc:407] Connection header received
0927 00:47:51.870779 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:51.870782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:51.870844 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:51.870931 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:51.871496 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:51.872090 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:51.872993 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:51.873251 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:51.873352 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:51.876864 (+ 3512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47115: BlockingRecv error: recv got EOF from 127.0.98.2:47115 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 00:47:51.979358 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 9152
I20260927 00:47:51.986502 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 9304
I20260927 00:47:51.993129 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 9436
I20260927 00:47:52.000135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 9082
2026-09-27T00:47:52Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18529 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T00:47:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:47:52Z Disabled control of system clock
I20260927 00:47:52.036015 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38923
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 00:47:52.179762 9632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:52.180184 9632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:52.180248 9632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:52.184993 9632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:47:52.185087 9632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:52.185105 9632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:52.185124 9632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:47:52.185142 9632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:47:52.190608 9632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38923
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:52.192212 9632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:52.194329 9632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:52.205861 9640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:52.205861 9637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:52.205941 9632 server_base.cc:1034] running on GCE node
W20260927 00:47:52.205861 9638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:52.207172 9632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:52.209064 9632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:52.210317 9632 hybrid_clock.cc:648] HybridClock initialized: now 1790470072210263 us; error 80 us; skew 500 ppm
I20260927 00:47:52.213318 9632 webserver.cc:460] Webserver started at http://127.0.98.62:36419/ using document root <none> and password file <none>
I20260927 00:47:52.214207 9632 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:52.214326 9632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:52.214679 9632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:52.217715 9632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "d23334ffd1794e788651bc001512a5d3"
format_stamp: "Formatted at 2026-09-27 00:47:52 on dist-test-slave-86rs"
server_key: "47af2055b220a77a3514993adefffaeb"
server_key_iv: "2830ab2dcaac3e6183f7617773c80e82"
server_key_version: "encryptionkey@0"
I20260927 00:47:52.218557 9632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "d23334ffd1794e788651bc001512a5d3"
format_stamp: "Formatted at 2026-09-27 00:47:52 on dist-test-slave-86rs"
server_key: "47af2055b220a77a3514993adefffaeb"
server_key_iv: "2830ab2dcaac3e6183f7617773c80e82"
server_key_version: "encryptionkey@0"
I20260927 00:47:52.223691 9632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 00:47:52.227540 9646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:52.229707 9632 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:47:52.229946 9632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "d23334ffd1794e788651bc001512a5d3"
format_stamp: "Formatted at 2026-09-27 00:47:52 on dist-test-slave-86rs"
server_key: "47af2055b220a77a3514993adefffaeb"
server_key_iv: "2830ab2dcaac3e6183f7617773c80e82"
server_key_version: "encryptionkey@0"
I20260927 00:47:52.230161 9632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:52.244436 9632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:52.245486 9632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:52.245748 9632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:52.255057 9632 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38923
I20260927 00:47:52.255069 9698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38923 every 8 connection(s)
I20260927 00:47:52.256482 9632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:47:52.260989 9699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:52.264552 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 9632
I20260927 00:47:52.264710 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:47:52.265074 392 external_mini_cluster.cc:1428] Setting key 6d850a7f980a8d501f3eb310f4d5d0c1
I20260927 00:47:52.269384 9699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d23334ffd1794e788651bc001512a5d3: Bootstrap starting.
I20260927 00:47:52.272416 9699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d23334ffd1794e788651bc001512a5d3: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:52.273393 9699 log.cc:824] T 00000000000000000000000000000000 P d23334ffd1794e788651bc001512a5d3: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:52.276098 9699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d23334ffd1794e788651bc001512a5d3: No bootstrap required, opened a new log
I20260927 00:47:52.277001 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:52.266995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45852 (local address 127.0.98.62:38923)
0927 00:47:52.267856 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:52.267876 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:52.267924 (+ 48us) server_negotiation.cc:407] Connection header received
0927 00:47:52.268825 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:52.268852 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:52.269526 (+ 674us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:52.270023 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:52.270612 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:52.271816 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:52.272989 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:52.273611 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:52.273818 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:52.274773 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:52.274779 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:52.274795 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:47:52.275270 (+ 475us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:52.275333 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:47:52.275354 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:52.275439 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:52.275625 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 00:47:52.275978 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":168,"threads_started":1}
I20260927 00:47:52.281144 9699 raft_consensus.cc:348] T 00000000000000000000000000000000 P d23334ffd1794e788651bc001512a5d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23334ffd1794e788651bc001512a5d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } }
I20260927 00:47:52.281430 9699 raft_consensus.cc:374] T 00000000000000000000000000000000 P d23334ffd1794e788651bc001512a5d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:52.281497 9699 raft_consensus.cc:729] T 00000000000000000000000000000000 P d23334ffd1794e788651bc001512a5d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d23334ffd1794e788651bc001512a5d3, State: Initialized, Role: FOLLOWER
I20260927 00:47:52.282084 9699 consensus_queue.cc:260] T 00000000000000000000000000000000 P d23334ffd1794e788651bc001512a5d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23334ffd1794e788651bc001512a5d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } }
I20260927 00:47:52.282232 9699 raft_consensus.cc:388] T 00000000000000000000000000000000 P d23334ffd1794e788651bc001512a5d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:47:52.282320 9699 raft_consensus.cc:482] T 00000000000000000000000000000000 P d23334ffd1794e788651bc001512a5d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:47:52.282437 9699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d23334ffd1794e788651bc001512a5d3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:52.283615 9699 raft_consensus.cc:504] T 00000000000000000000000000000000 P d23334ffd1794e788651bc001512a5d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23334ffd1794e788651bc001512a5d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } }
I20260927 00:47:52.284060 9699 leader_election.cc:302] T 00000000000000000000000000000000 P d23334ffd1794e788651bc001512a5d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d23334ffd1794e788651bc001512a5d3; no voters:
I20260927 00:47:52.284524 9699 leader_election.cc:288] T 00000000000000000000000000000000 P d23334ffd1794e788651bc001512a5d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:47:52.284662 9704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d23334ffd1794e788651bc001512a5d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:52.284912 9704 raft_consensus.cc:686] T 00000000000000000000000000000000 P d23334ffd1794e788651bc001512a5d3 [term 1 LEADER]: Becoming Leader. State: Replica: d23334ffd1794e788651bc001512a5d3, State: Running, Role: LEADER
I20260927 00:47:52.285338 9704 consensus_queue.cc:237] T 00000000000000000000000000000000 P d23334ffd1794e788651bc001512a5d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23334ffd1794e788651bc001512a5d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } }
I20260927 00:47:52.286419 9699 sys_catalog.cc:564] T 00000000000000000000000000000000 P d23334ffd1794e788651bc001512a5d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:47:52.287623 9705 sys_catalog.cc:455] T 00000000000000000000000000000000 P d23334ffd1794e788651bc001512a5d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d23334ffd1794e788651bc001512a5d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23334ffd1794e788651bc001512a5d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } } }
I20260927 00:47:52.287801 9705 sys_catalog.cc:458] T 00000000000000000000000000000000 P d23334ffd1794e788651bc001512a5d3 [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:52.288265 9713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:47:52.288205 9706 sys_catalog.cc:455] T 00000000000000000000000000000000 P d23334ffd1794e788651bc001512a5d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d23334ffd1794e788651bc001512a5d3. Latest consensus state: current_term: 1 leader_uuid: "d23334ffd1794e788651bc001512a5d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23334ffd1794e788651bc001512a5d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } } }
I20260927 00:47:52.288452 9706 sys_catalog.cc:458] T 00000000000000000000000000000000 P d23334ffd1794e788651bc001512a5d3 [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:52.292024 9713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:47:52.298962 9713 catalog_manager.cc:1347] Generated new cluster ID: 71e85f4c6e0a47ec9d9928956f1c053f
I20260927 00:47:52.299070 9713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:47:52.309799 9713 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:47:52.311039 9713 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:47:52.318163 9713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d23334ffd1794e788651bc001512a5d3: Generated new TSK 0
I20260927 00:47:52.319211 9713 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:47:52.329075 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38923
--builtin_ntp_servers=127.0.98.20:45281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 00:47:52.472611 9723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:52.473028 9723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:52.473116 9723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:52.477563 9723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:52.477671 9723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:52.477774 9723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:47:52.483217 9723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:52.484879 9723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:52.486927 9723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:52.501873 9729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:52.501871 9731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:52.501900 9728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:52.502377 9723 server_base.cc:1034] running on GCE node
I20260927 00:47:52.502919 9723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:52.503799 9723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:52.504994 9723 hybrid_clock.cc:648] HybridClock initialized: now 1790470072504965 us; error 45 us; skew 500 ppm
I20260927 00:47:52.507453 9723 webserver.cc:460] Webserver started at http://127.0.98.1:41493/ using document root <none> and password file <none>
I20260927 00:47:52.508365 9723 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:52.508502 9723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:52.508782 9723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:52.512310 9723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/data/instance:
uuid: "684dccf4d6ed48cf97bff4d4d0e7143e"
format_stamp: "Formatted at 2026-09-27 00:47:52 on dist-test-slave-86rs"
server_key: "8f63fb0cf2df4c5f474b215d30eb0fd3"
server_key_iv: "2bb722d7e30fc3eeee03756cab7a96d1"
server_key_version: "encryptionkey@0"
I20260927 00:47:52.513443 9723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/wal/instance:
uuid: "684dccf4d6ed48cf97bff4d4d0e7143e"
format_stamp: "Formatted at 2026-09-27 00:47:52 on dist-test-slave-86rs"
server_key: "8f63fb0cf2df4c5f474b215d30eb0fd3"
server_key_iv: "2bb722d7e30fc3eeee03756cab7a96d1"
server_key_version: "encryptionkey@0"
I20260927 00:47:52.519234 9723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 00:47:52.523145 9737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:52.525274 9723 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:47:52.525475 9723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "684dccf4d6ed48cf97bff4d4d0e7143e"
format_stamp: "Formatted at 2026-09-27 00:47:52 on dist-test-slave-86rs"
server_key: "8f63fb0cf2df4c5f474b215d30eb0fd3"
server_key_iv: "2bb722d7e30fc3eeee03756cab7a96d1"
server_key_version: "encryptionkey@0"
I20260927 00:47:52.525669 9723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:52.543098 9723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:52.544096 9723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:52.544382 9723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:52.545163 9723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:52.546372 9723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:52.546446 9723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:52.546541 9723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:52.546591 9723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:52.556933 9723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33369
I20260927 00:47:52.556965 9850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33369 every 8 connection(s)
I20260927 00:47:52.558101 9723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:47:52.558622 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 9723
I20260927 00:47:52.558751 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:47:52.559149 392 external_mini_cluster.cc:1428] Setting key a549d126d8f566756d610b771ac125f9
I20260927 00:47:52.561712 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38923
--builtin_ntp_servers=127.0.98.20:45281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 00:47:52.571372 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:52.560699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41431 (local address 127.0.98.62:38923)
0927 00:47:52.560847 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:52.560852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:52.562499 (+ 1647us) server_negotiation.cc:407] Connection header received
0927 00:47:52.564698 (+ 2199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:52.564701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:52.564798 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:52.564917 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:52.566129 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:52.567153 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:52.568648 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:52.569086 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:52.569236 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:52.570633 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:52.570640 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:52.570645 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:47:52.570666 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:52.570698 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:47:52.570711 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:52.570807 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:52.571020 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 00:47:52.571113 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:47:52.572568 9853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:52.561390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38923 (local address 127.0.98.1:41431)
0927 00:47:52.562308 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:52.562372 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:52.564363 (+ 1991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:52.565193 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:52.565203 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:52.565321 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:52.565965 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:52.565981 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:52.567310 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:52.567313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:52.568517 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:52.568528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:52.569267 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:52.569270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:52.569441 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:52.570277 (+ 836us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:47:52.570279 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:52.570333 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:47:52.570437 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:52.570834 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:52.570850 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:52.570856 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:47:52.570993 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 00:47:52.571332 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":165,"threads_started":1}
I20260927 00:47:52.574273 9851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38923
I20260927 00:47:52.574594 9851 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:52.575287 9851 heartbeater.cc:511] Master 127.0.98.62:38923 requested a full tablet report, sending...
I20260927 00:47:52.577324 9663 ts_manager.cc:194] Registered new tserver with Master: 684dccf4d6ed48cf97bff4d4d0e7143e (127.0.98.1:33369)
I20260927 00:47:52.578704 9663 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:41431
W20260927 00:47:52.701005 9854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:52.701390 9854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:52.701432 9854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:52.706496 9854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:52.706573 9854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:52.706672 9854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:47:52.711969 9854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:52.713611 9854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:52.715453 9854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:47:52.727674 9854 server_base.cc:1034] running on GCE node
W20260927 00:47:52.727603 9860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:52.727603 9859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:52.727603 9862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:52.728605 9854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:52.729596 9854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:52.730803 9854 hybrid_clock.cc:648] HybridClock initialized: now 1790470072730757 us; error 77 us; skew 500 ppm
I20260927 00:47:52.733390 9854 webserver.cc:460] Webserver started at http://127.0.98.2:44583/ using document root <none> and password file <none>
I20260927 00:47:52.734195 9854 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:52.734306 9854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:52.734642 9854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:52.737573 9854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/data/instance:
uuid: "a0c8c79180fc45dd8cdc6599025cb84e"
format_stamp: "Formatted at 2026-09-27 00:47:52 on dist-test-slave-86rs"
server_key: "fb28f3146dad7973e4ef6a20c9b63ab0"
server_key_iv: "68732bce9f7c9dccd1a79fc924086c62"
server_key_version: "encryptionkey@0"
I20260927 00:47:52.738330 9854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a0c8c79180fc45dd8cdc6599025cb84e"
format_stamp: "Formatted at 2026-09-27 00:47:52 on dist-test-slave-86rs"
server_key: "fb28f3146dad7973e4ef6a20c9b63ab0"
server_key_iv: "68732bce9f7c9dccd1a79fc924086c62"
server_key_version: "encryptionkey@0"
I20260927 00:47:52.743324 9854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 00:47:52.746688 9868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:52.748522 9854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:47:52.748677 9854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "a0c8c79180fc45dd8cdc6599025cb84e"
format_stamp: "Formatted at 2026-09-27 00:47:52 on dist-test-slave-86rs"
server_key: "fb28f3146dad7973e4ef6a20c9b63ab0"
server_key_iv: "68732bce9f7c9dccd1a79fc924086c62"
server_key_version: "encryptionkey@0"
I20260927 00:47:52.748821 9854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:52.766176 9854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:52.767175 9854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:52.767418 9854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:52.768191 9854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:52.769482 9854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:52.769558 9854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:52.769640 9854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:52.769691 9854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:52.780997 9854 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43471
I20260927 00:47:52.781018 9981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43471 every 8 connection(s)
I20260927 00:47:52.782248 9854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:47:52.790905 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 9854
I20260927 00:47:52.791050 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:47:52.791390 392 external_mini_cluster.cc:1428] Setting key d102d93e47875359cec5400ae39c109a
I20260927 00:47:52.793959 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38923
--builtin_ntp_servers=127.0.98.20:45281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 00:47:52.794878 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:52.784875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48659 (local address 127.0.98.62:38923)
0927 00:47:52.785088 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:52.785106 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:52.786656 (+ 1550us) server_negotiation.cc:407] Connection header received
0927 00:47:52.788614 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:52.788618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:52.788688 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:52.788775 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:52.789988 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:52.790646 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:52.792282 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:52.792555 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:52.792696 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:52.794307 (+ 1611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:52.794311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:52.794313 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:47:52.794327 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:52.794350 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:47:52.794358 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:52.794446 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:52.794632 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 00:47:52.794712 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 00:47:52.796172 9984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:52.785550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38923 (local address 127.0.98.2:48659)
0927 00:47:52.786474 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:52.786524 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:52.788333 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:52.789058 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:52.789068 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:52.789187 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:52.789821 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:52.789835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:52.790790 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:52.790793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:52.792058 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:52.792069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:52.792745 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:52.792749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:52.793004 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:52.793876 (+ 872us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:47:52.793878 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:52.793932 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:47:52.794024 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:52.794452 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:52.794464 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:52.794472 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 00:47:52.794636 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 00:47:52.795055 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":139,"threads_started":1}
I20260927 00:47:52.797787 9982 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38923
I20260927 00:47:52.798173 9982 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:52.798921 9982 heartbeater.cc:511] Master 127.0.98.62:38923 requested a full tablet report, sending...
I20260927 00:47:52.800320 9663 ts_manager.cc:194] Registered new tserver with Master: a0c8c79180fc45dd8cdc6599025cb84e (127.0.98.2:43471)
I20260927 00:47:52.801034 9663 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:48659
W20260927 00:47:52.934729 9985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:52.935161 9985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:52.935205 9985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:52.939682 9985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:52.939759 9985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:52.939823 9985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:47:52.945091 9985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:52.946595 9985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:52.948480 9985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:52.960734 9991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:52.960747 9990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:52.960770 9993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:52.961303 9985 server_base.cc:1034] running on GCE node
I20260927 00:47:52.961858 9985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:52.962704 9985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:52.963879 9985 hybrid_clock.cc:648] HybridClock initialized: now 1790470072963843 us; error 59 us; skew 500 ppm
I20260927 00:47:52.966874 9985 webserver.cc:460] Webserver started at http://127.0.98.3:45111/ using document root <none> and password file <none>
I20260927 00:47:52.967691 9985 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:52.967768 9985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:52.968030 9985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:52.970880 9985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/data/instance:
uuid: "5aa9e855f75640a2a5346cea6b9c0e29"
format_stamp: "Formatted at 2026-09-27 00:47:52 on dist-test-slave-86rs"
server_key: "da8d9e6322c5be875f6ccca0ff65a980"
server_key_iv: "98870c182e92e38f3c54c85933526d6e"
server_key_version: "encryptionkey@0"
I20260927 00:47:52.971571 9985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5aa9e855f75640a2a5346cea6b9c0e29"
format_stamp: "Formatted at 2026-09-27 00:47:52 on dist-test-slave-86rs"
server_key: "da8d9e6322c5be875f6ccca0ff65a980"
server_key_iv: "98870c182e92e38f3c54c85933526d6e"
server_key_version: "encryptionkey@0"
I20260927 00:47:52.977223 9985 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 00:47:52.981364 9999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:52.983486 9985 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:47:52.983651 9985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "5aa9e855f75640a2a5346cea6b9c0e29"
format_stamp: "Formatted at 2026-09-27 00:47:52 on dist-test-slave-86rs"
server_key: "da8d9e6322c5be875f6ccca0ff65a980"
server_key_iv: "98870c182e92e38f3c54c85933526d6e"
server_key_version: "encryptionkey@0"
I20260927 00:47:52.983805 9985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:52.996402 9985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:52.997512 9985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:52.997733 9985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:52.998493 9985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:52.999720 9985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:52.999774 9985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:52.999822 9985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:52.999837 9985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:53.012039 9985 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34957
I20260927 00:47:53.012022 10112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34957 every 8 connection(s)
I20260927 00:47:53.013633 9985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:47:53.023236 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 9985
I20260927 00:47:53.023397 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:47:53.023790 392 external_mini_cluster.cc:1428] Setting key f0a7b44908ef94ad7546e68ad54f83aa
I20260927 00:47:53.027014 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:53.016398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56873 (local address 127.0.98.62:38923)
0927 00:47:53.016585 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:53.016589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:53.018336 (+ 1747us) server_negotiation.cc:407] Connection header received
0927 00:47:53.020117 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:53.020121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:53.020200 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:53.020373 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:53.021862 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:53.022517 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:53.024508 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:53.024779 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:53.024915 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:53.026383 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:53.026386 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:53.026388 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:47:53.026401 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:53.026452 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:47:53.026464 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:53.026544 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:53.026747 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 00:47:53.026821 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 00:47:53.028395 10115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:53.017103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38923 (local address 127.0.98.3:56873)
0927 00:47:53.018109 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:53.018169 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:53.019789 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:53.020660 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:53.020670 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:53.020826 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:53.021657 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:53.021676 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:53.022669 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:53.022672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:53.024350 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:53.024364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:53.024996 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:53.025001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:53.025215 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:53.026035 (+ 820us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:47:53.026037 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:53.026096 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:47:53.026193 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:53.026578 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:53.026587 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:53.026593 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:47:53.026737 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 00:47:53.027130 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":160,"threads_started":1}
I20260927 00:47:53.029902 10113 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38923
I20260927 00:47:53.030260 10113 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:53.031106 10113 heartbeater.cc:511] Master 127.0.98.62:38923 requested a full tablet report, sending...
I20260927 00:47:53.032593 9663 ts_manager.cc:194] Registered new tserver with Master: 5aa9e855f75640a2a5346cea6b9c0e29 (127.0.98.3:34957)
I20260927 00:47:53.033401 9663 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:56873
I20260927 00:47:53.038736 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 00:47:53.042634 9702 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:53.041362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45860 (local address 127.0.98.62:38923)
0927 00:47:53.041574 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:53.041579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:53.041780 (+ 201us) server_negotiation.cc:407] Connection header received
0927 00:47:53.041955 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:53.041959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:53.042026 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:53.042120 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:53.042437 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45860: BlockingRecv error: recv got EOF from 127.0.0.1:45860 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 00:47:53.042838 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 9723
I20260927 00:47:53.050001 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 9854
I20260927 00:47:53.056097 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 9985
I20260927 00:47:53.062594 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 9632
2026-09-27T00:47:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (1063 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T00:47:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:47:53Z Disabled control of system clock
I20260927 00:47:53.100862 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41551
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 00:47:53.249382 10124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:53.249847 10124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:53.249933 10124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:53.255793 10124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:47:53.255985 10124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:53.256021 10124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:53.256054 10124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:47:53.256103 10124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:47:53.261946 10124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41551
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:53.263552 10124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:53.265477 10124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:53.276602 10132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:53.276594 10129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:53.276767 10130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:53.277429 10124 server_base.cc:1034] running on GCE node
I20260927 00:47:53.278374 10124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:53.280100 10124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:53.281347 10124 hybrid_clock.cc:648] HybridClock initialized: now 1790470073281321 us; error 55 us; skew 500 ppm
I20260927 00:47:53.290218 10124 webserver.cc:460] Webserver started at http://127.0.98.62:42267/ using document root <none> and password file <none>
I20260927 00:47:53.291218 10124 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:53.291352 10124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:53.291687 10124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:53.294858 10124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "be75a463cecf4f0fa37f86b8682fc749"
format_stamp: "Formatted at 2026-09-27 00:47:53 on dist-test-slave-86rs"
server_key: "7a1b9836c7b6c2ff796d39ff2b0261a2"
server_key_iv: "cf3329ece996b8e3b522b249f6d22ac9"
server_key_version: "encryptionkey@0"
I20260927 00:47:53.295657 10124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "be75a463cecf4f0fa37f86b8682fc749"
format_stamp: "Formatted at 2026-09-27 00:47:53 on dist-test-slave-86rs"
server_key: "7a1b9836c7b6c2ff796d39ff2b0261a2"
server_key_iv: "cf3329ece996b8e3b522b249f6d22ac9"
server_key_version: "encryptionkey@0"
I20260927 00:47:53.301327 10124 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 00:47:53.305045 10138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:53.306936 10124 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:47:53.307155 10124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "be75a463cecf4f0fa37f86b8682fc749"
format_stamp: "Formatted at 2026-09-27 00:47:53 on dist-test-slave-86rs"
server_key: "7a1b9836c7b6c2ff796d39ff2b0261a2"
server_key_iv: "cf3329ece996b8e3b522b249f6d22ac9"
server_key_version: "encryptionkey@0"
I20260927 00:47:53.307337 10124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:53.313964 10124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:53.314828 10124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:53.315075 10124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:53.324714 10124 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41551
I20260927 00:47:53.324738 10190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41551 every 8 connection(s)
I20260927 00:47:53.326242 10124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:47:53.329722 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 10124
I20260927 00:47:53.329986 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:47:53.330375 392 external_mini_cluster.cc:1428] Setting key 5031b21ced9ce8d5534713d501284b88
I20260927 00:47:53.331421 10191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:53.338918 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:53.332564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55402 (local address 127.0.98.62:41551)
0927 00:47:53.333589 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:53.333612 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:53.333668 (+ 56us) server_negotiation.cc:407] Connection header received
0927 00:47:53.334599 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:53.334626 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:53.335227 (+ 601us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:53.335783 (+ 556us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:53.336953 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:53.336958 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:53.336982 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:47:53.336994 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:53.337077 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:47:53.337099 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:53.337324 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:53.337418 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 00:47:53.337787 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":210,"threads_started":1}
I20260927 00:47:53.341367 10191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be75a463cecf4f0fa37f86b8682fc749: Bootstrap starting.
I20260927 00:47:53.345119 10191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be75a463cecf4f0fa37f86b8682fc749: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:53.346117 10191 log.cc:824] T 00000000000000000000000000000000 P be75a463cecf4f0fa37f86b8682fc749: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:53.348946 10191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be75a463cecf4f0fa37f86b8682fc749: No bootstrap required, opened a new log
I20260927 00:47:53.354089 10191 raft_consensus.cc:348] T 00000000000000000000000000000000 P be75a463cecf4f0fa37f86b8682fc749 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be75a463cecf4f0fa37f86b8682fc749" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41551 } }
I20260927 00:47:53.354499 10191 raft_consensus.cc:374] T 00000000000000000000000000000000 P be75a463cecf4f0fa37f86b8682fc749 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:53.354573 10191 raft_consensus.cc:729] T 00000000000000000000000000000000 P be75a463cecf4f0fa37f86b8682fc749 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be75a463cecf4f0fa37f86b8682fc749, State: Initialized, Role: FOLLOWER
I20260927 00:47:53.355327 10191 consensus_queue.cc:260] T 00000000000000000000000000000000 P be75a463cecf4f0fa37f86b8682fc749 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be75a463cecf4f0fa37f86b8682fc749" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41551 } }
I20260927 00:47:53.355540 10191 raft_consensus.cc:388] T 00000000000000000000000000000000 P be75a463cecf4f0fa37f86b8682fc749 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:47:53.355614 10191 raft_consensus.cc:482] T 00000000000000000000000000000000 P be75a463cecf4f0fa37f86b8682fc749 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:47:53.355710 10191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be75a463cecf4f0fa37f86b8682fc749 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:53.357041 10191 raft_consensus.cc:504] T 00000000000000000000000000000000 P be75a463cecf4f0fa37f86b8682fc749 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be75a463cecf4f0fa37f86b8682fc749" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41551 } }
I20260927 00:47:53.357465 10191 leader_election.cc:302] T 00000000000000000000000000000000 P be75a463cecf4f0fa37f86b8682fc749 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be75a463cecf4f0fa37f86b8682fc749; no voters:
I20260927 00:47:53.357822 10191 leader_election.cc:288] T 00000000000000000000000000000000 P be75a463cecf4f0fa37f86b8682fc749 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:47:53.357975 10196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be75a463cecf4f0fa37f86b8682fc749 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:53.358273 10196 raft_consensus.cc:686] T 00000000000000000000000000000000 P be75a463cecf4f0fa37f86b8682fc749 [term 1 LEADER]: Becoming Leader. State: Replica: be75a463cecf4f0fa37f86b8682fc749, State: Running, Role: LEADER
I20260927 00:47:53.358721 10196 consensus_queue.cc:237] T 00000000000000000000000000000000 P be75a463cecf4f0fa37f86b8682fc749 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be75a463cecf4f0fa37f86b8682fc749" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41551 } }
I20260927 00:47:53.359553 10191 sys_catalog.cc:564] T 00000000000000000000000000000000 P be75a463cecf4f0fa37f86b8682fc749 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:47:53.360977 10198 sys_catalog.cc:455] T 00000000000000000000000000000000 P be75a463cecf4f0fa37f86b8682fc749 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be75a463cecf4f0fa37f86b8682fc749. Latest consensus state: current_term: 1 leader_uuid: "be75a463cecf4f0fa37f86b8682fc749" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be75a463cecf4f0fa37f86b8682fc749" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41551 } } }
I20260927 00:47:53.361127 10198 sys_catalog.cc:458] T 00000000000000000000000000000000 P be75a463cecf4f0fa37f86b8682fc749 [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:53.361452 10197 sys_catalog.cc:455] T 00000000000000000000000000000000 P be75a463cecf4f0fa37f86b8682fc749 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be75a463cecf4f0fa37f86b8682fc749" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be75a463cecf4f0fa37f86b8682fc749" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41551 } } }
I20260927 00:47:53.361572 10197 sys_catalog.cc:458] T 00000000000000000000000000000000 P be75a463cecf4f0fa37f86b8682fc749 [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:53.361613 10205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:47:53.365806 10205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:47:53.374138 10205 catalog_manager.cc:1347] Generated new cluster ID: 08142d81a5964b138729660981d2bb9a
I20260927 00:47:53.374266 10205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:47:53.393365 10205 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:47:53.393923 10205 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:47:53.403389 10205 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be75a463cecf4f0fa37f86b8682fc749: Generated new TSK 0
I20260927 00:47:53.404628 10205 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:47:53.416755 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41551
--builtin_ntp_servers=127.0.98.20:44917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 00:47:53.561395 10215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:53.561830 10215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:53.561919 10215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:53.566569 10215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:53.566663 10215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:53.566732 10215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:47:53.572129 10215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:53.573715 10215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:53.575675 10215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:53.587417 10223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:53.587459 10220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:53.587411 10221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:53.587661 10215 server_base.cc:1034] running on GCE node
I20260927 00:47:53.588392 10215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:53.589267 10215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:53.590441 10215 hybrid_clock.cc:648] HybridClock initialized: now 1790470073590418 us; error 43 us; skew 500 ppm
I20260927 00:47:53.592865 10215 webserver.cc:460] Webserver started at http://127.0.98.1:39039/ using document root <none> and password file <none>
I20260927 00:47:53.593629 10215 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:53.593700 10215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:53.593967 10215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:53.596863 10215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/data/instance:
uuid: "cb633d50e2bb4c10b54963b45731a56c"
format_stamp: "Formatted at 2026-09-27 00:47:53 on dist-test-slave-86rs"
server_key: "fa34c2fb99d7a24e3bb06514175ba030"
server_key_iv: "33ea7470091842be63701a806936550f"
server_key_version: "encryptionkey@0"
I20260927 00:47:53.597582 10215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cb633d50e2bb4c10b54963b45731a56c"
format_stamp: "Formatted at 2026-09-27 00:47:53 on dist-test-slave-86rs"
server_key: "fa34c2fb99d7a24e3bb06514175ba030"
server_key_iv: "33ea7470091842be63701a806936550f"
server_key_version: "encryptionkey@0"
I20260927 00:47:53.602707 10215 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 00:47:53.606380 10229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:53.608377 10215 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:47:53.608606 10215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "cb633d50e2bb4c10b54963b45731a56c"
format_stamp: "Formatted at 2026-09-27 00:47:53 on dist-test-slave-86rs"
server_key: "fa34c2fb99d7a24e3bb06514175ba030"
server_key_iv: "33ea7470091842be63701a806936550f"
server_key_version: "encryptionkey@0"
I20260927 00:47:53.608785 10215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:53.615499 10215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:53.616395 10215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:53.616639 10215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:53.617357 10215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:53.618523 10215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:53.618597 10215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:53.618677 10215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:53.618727 10215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:53.629567 10215 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42337
I20260927 00:47:53.629585 10342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42337 every 8 connection(s)
I20260927 00:47:53.630841 10215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:47:53.635869 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 10215
I20260927 00:47:53.636082 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:47:53.636526 392 external_mini_cluster.cc:1428] Setting key d01ee8d1b3fd8864119a4f3e3d718a1a
I20260927 00:47:53.639063 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41551
--builtin_ntp_servers=127.0.98.20:44917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 00:47:53.643533 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:53.633609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40237 (local address 127.0.98.62:41551)
0927 00:47:53.633809 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:53.633815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:53.635475 (+ 1660us) server_negotiation.cc:407] Connection header received
0927 00:47:53.637291 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:53.637295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:53.637349 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:53.637431 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:53.642862 (+ 5431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:53.642867 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:53.642880 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:47:53.642907 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:53.642939 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:47:53.642950 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:53.643083 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:53.643281 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 00:47:53.643350 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 00:47:53.645030 10345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:53.634293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41551 (local address 127.0.98.1:40237)
0927 00:47:53.635270 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:53.635326 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:53.636986 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:53.637686 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:53.637695 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:53.637823 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:53.642367 (+ 4544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:47:53.642369 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:53.642465 (+ 96us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:47:53.642597 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:53.643057 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:53.643070 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:53.643078 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 00:47:53.643349 (+ 271us) client_negotiation.cc:240] Negotiation successful
0927 00:47:53.643787 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":189,"threads_started":1}
I20260927 00:47:53.646986 10343 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41551
I20260927 00:47:53.647332 10343 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:53.647931 10343 heartbeater.cc:511] Master 127.0.98.62:41551 requested a full tablet report, sending...
I20260927 00:47:53.650135 10155 ts_manager.cc:194] Registered new tserver with Master: cb633d50e2bb4c10b54963b45731a56c (127.0.98.1:42337)
W20260927 00:47:53.779728 10346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:53.780189 10346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:53.780296 10346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:53.785076 10346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:53.785171 10346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:53.785267 10346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:47:53.790825 10346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:53.792591 10346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:53.794574 10346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:53.807339 10351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:53.807346 10352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:53.807353 10354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:53.807451 10346 server_base.cc:1034] running on GCE node
I20260927 00:47:53.808203 10346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:53.809118 10346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:53.810313 10346 hybrid_clock.cc:648] HybridClock initialized: now 1790470073810282 us; error 49 us; skew 500 ppm
I20260927 00:47:53.813521 10346 webserver.cc:460] Webserver started at http://127.0.98.2:36029/ using document root <none> and password file <none>
I20260927 00:47:53.814363 10346 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:53.814442 10346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:53.814663 10346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:53.817426 10346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/data/instance:
uuid: "d8c516ef3e444bc8b444f61f4fdea6c0"
format_stamp: "Formatted at 2026-09-27 00:47:53 on dist-test-slave-86rs"
server_key: "a0c0018aa9aff25c6fda7aa85dee2b61"
server_key_iv: "65141742a1f7e32c3efed0625c018843"
server_key_version: "encryptionkey@0"
I20260927 00:47:53.818148 10346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d8c516ef3e444bc8b444f61f4fdea6c0"
format_stamp: "Formatted at 2026-09-27 00:47:53 on dist-test-slave-86rs"
server_key: "a0c0018aa9aff25c6fda7aa85dee2b61"
server_key_iv: "65141742a1f7e32c3efed0625c018843"
server_key_version: "encryptionkey@0"
I20260927 00:47:53.823187 10346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 00:47:53.826256 10360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:53.828455 10346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:47:53.828600 10346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "d8c516ef3e444bc8b444f61f4fdea6c0"
format_stamp: "Formatted at 2026-09-27 00:47:53 on dist-test-slave-86rs"
server_key: "a0c0018aa9aff25c6fda7aa85dee2b61"
server_key_iv: "65141742a1f7e32c3efed0625c018843"
server_key_version: "encryptionkey@0"
I20260927 00:47:53.828748 10346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:53.837087 10346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:53.837946 10346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:53.838186 10346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:53.838943 10346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:53.840440 10346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:53.840528 10346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:53.840606 10346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:53.840637 10346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:53.855890 10346 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36049
I20260927 00:47:53.855921 10473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36049 every 8 connection(s)
I20260927 00:47:53.857793 10346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:47:53.867311 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:53.860783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55871 (local address 127.0.98.62:41551)
0927 00:47:53.860967 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:53.860975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:53.862806 (+ 1831us) server_negotiation.cc:407] Connection header received
0927 00:47:53.864691 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:53.864695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:53.864757 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:53.864844 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:53.866631 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:53.866636 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:53.866639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:47:53.866655 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:53.866684 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:47:53.866696 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:53.866789 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:53.867063 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 00:47:53.867131 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:47:53.867800 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 10346
I20260927 00:47:53.867944 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:47:53.868377 392 external_mini_cluster.cc:1428] Setting key 8aea2ba08385d87645f0508277c4014b
I20260927 00:47:53.869884 10476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:53.861616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41551 (local address 127.0.98.2:55871)
0927 00:47:53.862630 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:53.862682 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:53.864405 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:53.865120 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:53.865132 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:53.865303 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:53.866127 (+ 824us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:47:53.866132 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:53.866223 (+ 91us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:47:53.866357 (+ 134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:53.866871 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:53.866879 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:53.866885 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:47:53.867118 (+ 233us) client_negotiation.cc:240] Negotiation successful
0927 00:47:53.868379 (+ 1261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":188,"threads_started":1}
I20260927 00:47:53.871359 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41551
--builtin_ntp_servers=127.0.98.20:44917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 00:47:53.874253 10474 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41551
I20260927 00:47:53.874673 10474 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:53.875430 10474 heartbeater.cc:511] Master 127.0.98.62:41551 requested a full tablet report, sending...
I20260927 00:47:53.877317 10155 ts_manager.cc:194] Registered new tserver with Master: d8c516ef3e444bc8b444f61f4fdea6c0 (127.0.98.2:36049)
W20260927 00:47:54.015930 10477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:54.016448 10477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:54.016547 10477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:54.021906 10477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:54.022075 10477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:54.022202 10477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:47:54.028584 10477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:54.030601 10477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:54.033316 10477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:54.046720 10483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:54.046720 10485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:54.046746 10482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:54.046978 10477 server_base.cc:1034] running on GCE node
I20260927 00:47:54.047755 10477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:54.048736 10477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:54.049965 10477 hybrid_clock.cc:648] HybridClock initialized: now 1790470074049920 us; error 66 us; skew 500 ppm
I20260927 00:47:54.052501 10477 webserver.cc:460] Webserver started at http://127.0.98.3:32827/ using document root <none> and password file <none>
I20260927 00:47:54.053346 10477 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:54.053450 10477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:54.053742 10477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:54.056557 10477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/data/instance:
uuid: "caf7c0e880f041babb63dc6291bbed17"
format_stamp: "Formatted at 2026-09-27 00:47:54 on dist-test-slave-86rs"
server_key: "4ff9ab7fa3e05d1cef7ca027651e66b4"
server_key_iv: "386f021e4366305a261f620954a172e1"
server_key_version: "encryptionkey@0"
I20260927 00:47:54.057286 10477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/wal/instance:
uuid: "caf7c0e880f041babb63dc6291bbed17"
format_stamp: "Formatted at 2026-09-27 00:47:54 on dist-test-slave-86rs"
server_key: "4ff9ab7fa3e05d1cef7ca027651e66b4"
server_key_iv: "386f021e4366305a261f620954a172e1"
server_key_version: "encryptionkey@0"
I20260927 00:47:54.062276 10477 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 00:47:54.065899 10491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:54.067679 10477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:47:54.067865 10477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "caf7c0e880f041babb63dc6291bbed17"
format_stamp: "Formatted at 2026-09-27 00:47:54 on dist-test-slave-86rs"
server_key: "4ff9ab7fa3e05d1cef7ca027651e66b4"
server_key_iv: "386f021e4366305a261f620954a172e1"
server_key_version: "encryptionkey@0"
I20260927 00:47:54.068043 10477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:54.074389 10477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:54.075218 10477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:54.075438 10477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:54.076135 10477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:54.077414 10477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:54.077499 10477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:54.077600 10477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:54.077642 10477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:54.088487 10477 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40599
I20260927 00:47:54.088538 10604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40599 every 8 connection(s)
I20260927 00:47:54.089785 10477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:47:54.091450 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 10477
I20260927 00:47:54.091568 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790470006085796-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:47:54.091996 392 external_mini_cluster.cc:1428] Setting key 65d3815589ca7736c5568a0d4f344c9e
I20260927 00:47:54.098613 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:54.092734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36793 (local address 127.0.98.62:41551)
0927 00:47:54.092943 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:54.092947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:54.094793 (+ 1846us) server_negotiation.cc:407] Connection header received
0927 00:47:54.096436 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:54.096439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:54.096492 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:54.096560 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:54.098016 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:54.098020 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:54.098023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:47:54.098038 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:54.098067 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:47:54.098085 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:54.098190 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:54.098384 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 00:47:54.098442 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 00:47:54.099833 10607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:54.093448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41551 (local address 127.0.98.3:36793)
0927 00:47:54.094592 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:54.094662 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:54.096119 (+ 1457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:54.096808 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:54.096818 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:54.096946 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:54.097694 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:47:54.097695 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:54.097751 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:47:54.097846 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:54.098173 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:54.098182 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:54.098188 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:47:54.098435 (+ 247us) client_negotiation.cc:240] Negotiation successful
0927 00:47:54.098775 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":228,"threads_started":1}
I20260927 00:47:54.101189 10605 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41551
I20260927 00:47:54.101449 10605 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:54.101990 10605 heartbeater.cc:511] Master 127.0.98.62:41551 requested a full tablet report, sending...
I20260927 00:47:54.103365 10155 ts_manager.cc:194] Registered new tserver with Master: caf7c0e880f041babb63dc6291bbed17 (127.0.98.3:40599)
I20260927 00:47:54.107009 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 00:47:54.110858 10194 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:47:54.109627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55406 (local address 127.0.98.62:41551)
0927 00:47:54.109773 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:54.109777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:54.110018 (+ 241us) server_negotiation.cc:407] Connection header received
0927 00:47:54.110124 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:54.110126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:54.110167 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:54.110242 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:54.110635 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55406: BlockingRecv error: recv got EOF from 127.0.0.1:55406 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:47:54.111045 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 10215
I20260927 00:47:54.118463 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 10346
I20260927 00:47:54.125057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 10477
I20260927 00:47:54.131301 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 10124
2026-09-27T00:47:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (1064 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:47:54 dist-test-slave-86rs krb5kdc[10615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:47:54 dist-test-slave-86rs krb5kdc[10615](info): set up 2 sockets
Sep 27 00:47:54 dist-test-slave-86rs krb5kdc[10615](info): commencing operation
krb5kdc: starting...
W20260927 00:47:55.189646 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:47:55 dist-test-slave-86rs krb5kdc[10615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:47:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:47:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:47:55.316934 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40475
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:55.443698 10628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:55.444077 10628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:55.444161 10628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:55.448468 10628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:47:55.448590 10628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:55.448625 10628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:55.448665 10628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:47:55.448695 10628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:47:55.454439 10628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:55.455699 10628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:55.456903 10628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:55.466387 10636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:55.466403 10633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:55.466396 10634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:55.466630 10628 server_base.cc:1034] running on GCE node
I20260927 00:47:55.467227 10628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:55.468569 10628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:55.469775 10628 hybrid_clock.cc:648] HybridClock initialized: now 1790470075469757 us; error 32 us; skew 500 ppm
Sep 27 00:47:55 dist-test-slave-86rs krb5kdc[10615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:55.472669 10628 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:47:55.473696 10628 webserver.cc:460] Webserver started at http://127.0.98.62:36321/ using document root <none> and password file <none>
I20260927 00:47:55.474270 10628 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:55.474329 10628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:55.474507 10628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:55.476714 10628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "86b4ffbb571a4973aa3738c7af8a6d78"
format_stamp: "Formatted at 2026-09-27 00:47:55 on dist-test-slave-86rs"
server_key: "231f5656b6b5c37bfef72fed5ad13d91"
server_key_iv: "236c635c27e2398455ed4dc16df98ad2"
server_key_version: "encryptionkey@0"
I20260927 00:47:55.477347 10628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "86b4ffbb571a4973aa3738c7af8a6d78"
format_stamp: "Formatted at 2026-09-27 00:47:55 on dist-test-slave-86rs"
server_key: "231f5656b6b5c37bfef72fed5ad13d91"
server_key_iv: "236c635c27e2398455ed4dc16df98ad2"
server_key_version: "encryptionkey@0"
I20260927 00:47:55.481443 10628 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:47:55.484093 10643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:55.485599 10628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:47:55.485755 10628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "86b4ffbb571a4973aa3738c7af8a6d78"
format_stamp: "Formatted at 2026-09-27 00:47:55 on dist-test-slave-86rs"
server_key: "231f5656b6b5c37bfef72fed5ad13d91"
server_key_iv: "236c635c27e2398455ed4dc16df98ad2"
server_key_version: "encryptionkey@0"
I20260927 00:47:55.485880 10628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:55.507683 10628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:55.511320 10628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:55.511565 10628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:55.519375 10628 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40475
I20260927 00:47:55.519693 10695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40475 every 8 connection(s)
I20260927 00:47:55.520505 10628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:47:55.523741 10696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:55.524187 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 10628
I20260927 00:47:55.524362 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:47:55.524705 392 external_mini_cluster.cc:1428] Setting key 09357c7c9c9fe951d4dd05c770fb17bb
I20260927 00:47:55.532307 10696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86b4ffbb571a4973aa3738c7af8a6d78: Bootstrap starting.
I20260927 00:47:55.535240 10696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86b4ffbb571a4973aa3738c7af8a6d78: Neither blocks nor log segments found. Creating new log.
Sep 27 00:47:55 dist-test-slave-86rs krb5kdc[10615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:55.536159 10696 log.cc:824] T 00000000000000000000000000000000 P 86b4ffbb571a4973aa3738c7af8a6d78: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:55.538966 10696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86b4ffbb571a4973aa3738c7af8a6d78: No bootstrap required, opened a new log
I20260927 00:47:55.541760 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:55.526302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56268 (local address 127.0.98.62:40475)
0927 00:47:55.526939 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:55.526959 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:55.527006 (+ 47us) server_negotiation.cc:407] Connection header received
0927 00:47:55.528051 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:55.528078 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:55.528589 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:55.529009 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:55.529770 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:55.531342 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:55.532376 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:55.533111 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:55.533276 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:55.536015 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:55.536024 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:55.536046 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:55.536080 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:55.538185 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:55.538673 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:55.538679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:55.538685 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:55.538841 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:55.539238 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:55.539241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:55.539249 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:55.539786 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:55.539997 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:55.540301 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 00:47:55.540630 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":130,"threads_started":1}
I20260927 00:47:55.543793 10696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86b4ffbb571a4973aa3738c7af8a6d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b4ffbb571a4973aa3738c7af8a6d78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40475 } }
I20260927 00:47:55.544081 10696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86b4ffbb571a4973aa3738c7af8a6d78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:55.544137 10696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86b4ffbb571a4973aa3738c7af8a6d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86b4ffbb571a4973aa3738c7af8a6d78, State: Initialized, Role: FOLLOWER
I20260927 00:47:55.544617 10696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86b4ffbb571a4973aa3738c7af8a6d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b4ffbb571a4973aa3738c7af8a6d78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40475 } }
I20260927 00:47:55.544755 10696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86b4ffbb571a4973aa3738c7af8a6d78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:47:55.544845 10696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86b4ffbb571a4973aa3738c7af8a6d78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:47:55.544951 10696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86b4ffbb571a4973aa3738c7af8a6d78 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:55.545929 10696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86b4ffbb571a4973aa3738c7af8a6d78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b4ffbb571a4973aa3738c7af8a6d78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40475 } }
I20260927 00:47:55.546303 10696 leader_election.cc:302] T 00000000000000000000000000000000 P 86b4ffbb571a4973aa3738c7af8a6d78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86b4ffbb571a4973aa3738c7af8a6d78; no voters:
I20260927 00:47:55.546658 10696 leader_election.cc:288] T 00000000000000000000000000000000 P 86b4ffbb571a4973aa3738c7af8a6d78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:47:55.546828 10701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86b4ffbb571a4973aa3738c7af8a6d78 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:55.547073 10701 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86b4ffbb571a4973aa3738c7af8a6d78 [term 1 LEADER]: Becoming Leader. State: Replica: 86b4ffbb571a4973aa3738c7af8a6d78, State: Running, Role: LEADER
I20260927 00:47:55.547400 10701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86b4ffbb571a4973aa3738c7af8a6d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b4ffbb571a4973aa3738c7af8a6d78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40475 } }
I20260927 00:47:55.548007 10696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86b4ffbb571a4973aa3738c7af8a6d78 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:47:55.549265 10702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86b4ffbb571a4973aa3738c7af8a6d78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86b4ffbb571a4973aa3738c7af8a6d78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b4ffbb571a4973aa3738c7af8a6d78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40475 } } }
I20260927 00:47:55.549412 10702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86b4ffbb571a4973aa3738c7af8a6d78 [sys.catalog]: This master's current role is: LEADER
I20260927 00:47:55.549779 10703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86b4ffbb571a4973aa3738c7af8a6d78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86b4ffbb571a4973aa3738c7af8a6d78. Latest consensus state: current_term: 1 leader_uuid: "86b4ffbb571a4973aa3738c7af8a6d78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b4ffbb571a4973aa3738c7af8a6d78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40475 } } }
I20260927 00:47:55.549868 10703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86b4ffbb571a4973aa3738c7af8a6d78 [sys.catalog]: This master's current role is: LEADER
W20260927 00:47:55.553346 10716 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 86b4ffbb571a4973aa3738c7af8a6d78: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:47:55.553418 10716 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:47:55.553531 10710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:47:55.554270 10710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:47:55.560609 10710 catalog_manager.cc:1347] Generated new cluster ID: 87c172c503654d6ca057b87550dedeea
I20260927 00:47:55.560714 10710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:47:55.594573 10710 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:47:55.595649 10710 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:47:55.604161 10710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86b4ffbb571a4973aa3738c7af8a6d78: Generated new TSK 0
I20260927 00:47:55.605137 10710 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:47:55.643772 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40475
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:55.760113 10722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:55.760499 10722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:55.760564 10722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:55.764859 10722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:55.764932 10722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:55.765026 10722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:47:55.770723 10722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:55.771890 10722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:55.773025 10722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:55.783193 10727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:55.783205 10730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:55.783212 10728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:55.783509 10722 server_base.cc:1034] running on GCE node
I20260927 00:47:55.784070 10722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:55.784850 10722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:55.786057 10722 hybrid_clock.cc:648] HybridClock initialized: now 1790470075786038 us; error 34 us; skew 500 ppm
Sep 27 00:47:55 dist-test-slave-86rs krb5kdc[10615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:55.789026 10722 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:47:55.790150 10722 webserver.cc:460] Webserver started at http://127.0.98.1:38807/ using document root <none> and password file <none>
I20260927 00:47:55.790812 10722 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:55.790907 10722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:55.791189 10722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:55.793530 10722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-0/data/instance:
uuid: "d6b096bba43c463194042a895634d6db"
format_stamp: "Formatted at 2026-09-27 00:47:55 on dist-test-slave-86rs"
server_key: "1ae0928ac2c06cb2c0c0d483e98c1eb2"
server_key_iv: "3359ded19ef5c54d168fa0e3e4f02994"
server_key_version: "encryptionkey@0"
I20260927 00:47:55.794174 10722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d6b096bba43c463194042a895634d6db"
format_stamp: "Formatted at 2026-09-27 00:47:55 on dist-test-slave-86rs"
server_key: "1ae0928ac2c06cb2c0c0d483e98c1eb2"
server_key_iv: "3359ded19ef5c54d168fa0e3e4f02994"
server_key_version: "encryptionkey@0"
I20260927 00:47:55.798485 10722 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:47:55.801474 10737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:55.802951 10722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:47:55.803125 10722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "d6b096bba43c463194042a895634d6db"
format_stamp: "Formatted at 2026-09-27 00:47:55 on dist-test-slave-86rs"
server_key: "1ae0928ac2c06cb2c0c0d483e98c1eb2"
server_key_iv: "3359ded19ef5c54d168fa0e3e4f02994"
server_key_version: "encryptionkey@0"
I20260927 00:47:55.803280 10722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:55.838424 10722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:55.841763 10722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:55.841941 10722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:55.842509 10722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:55.843338 10722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:55.843385 10722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:55.843425 10722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:55.843441 10722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:55.852820 10722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37543
I20260927 00:47:55.852856 10850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37543 every 8 connection(s)
I20260927 00:47:55.853789 10722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:47:55.861351 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 10722
I20260927 00:47:55.861480 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:47:55.861852 392 external_mini_cluster.cc:1428] Setting key 30cab8a0e8ea4698eaeafea9c3a63498
Sep 27 00:47:55 dist-test-slave-86rs krb5kdc[10615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:55.868804 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:55.855832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40951 (local address 127.0.98.62:40475)
0927 00:47:55.855984 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:55.855987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:55.857125 (+ 1138us) server_negotiation.cc:407] Connection header received
0927 00:47:55.858001 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:55.858004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:55.858054 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:55.858132 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:55.859401 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:55.859963 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:55.861268 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:55.861620 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:55.861744 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:55.864629 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:55.864636 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:55.864642 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:55.864677 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:55.866271 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:55.866973 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:55.866976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:55.866977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:55.867051 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:55.867468 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:55.867470 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:55.867471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:55.867718 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:55.867796 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:55.868557 (+ 761us) server_negotiation.cc:299] Negotiation successful
0927 00:47:55.868669 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:47:55.870193 10853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:55.856145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40475 (local address 127.0.98.1:40951)
0927 00:47:55.856972 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:55.857015 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:55.857790 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:55.858313 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:55.858319 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:55.858741 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:55.859245 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:55.859259 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:55.860087 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:55.860089 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:55.861125 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:55.861135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:55.861811 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:55.861814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:55.862033 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:55.862737 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:55.862754 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:55.864459 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:55.866404 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:55.866414 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:55.866436 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:55.866846 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:55.867148 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:55.867154 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:55.867159 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:55.867368 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:55.867836 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:55.867843 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:55.868262 (+ 419us) client_negotiation.cc:769] Sending connection context
0927 00:47:55.868511 (+ 249us) client_negotiation.cc:240] Negotiation successful
0927 00:47:55.868935 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":110,"threads_started":1}
I20260927 00:47:55.871578 10851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40475
I20260927 00:47:55.871857 10851 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:47:55.872452 10851 heartbeater.cc:511] Master 127.0.98.62:40475 requested a full tablet report, sending...
I20260927 00:47:55.874164 10660 ts_manager.cc:194] Registered new tserver with Master: d6b096bba43c463194042a895634d6db (127.0.98.1:37543)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:47:55.875566 10660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40951
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:47:55.891518 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40475
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:56.010108 10856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:56.010435 10856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:56.010483 10856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:56.015085 10856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:56.015189 10856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:56.015255 10856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:47:56.021229 10856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:56.022845 10856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:56.024348 10856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:56.036597 10864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:56.036628 10862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:56.036602 10861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:56.037063 10856 server_base.cc:1034] running on GCE node
I20260927 00:47:56.037529 10856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:56.038226 10856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:56.039422 10856 hybrid_clock.cc:648] HybridClock initialized: now 1790470076039401 us; error 35 us; skew 500 ppm
Sep 27 00:47:56 dist-test-slave-86rs krb5kdc[10615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:56.042451 10856 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:47:56.043535 10856 webserver.cc:460] Webserver started at http://127.0.98.2:43641/ using document root <none> and password file <none>
I20260927 00:47:56.044150 10856 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:56.044214 10856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:56.044468 10856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:56.046626 10856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-1/data/instance:
uuid: "dcc6819721594bd18ba45d1220ca8309"
format_stamp: "Formatted at 2026-09-27 00:47:56 on dist-test-slave-86rs"
server_key: "d2e47b1783a96ba24129034f70d0d582"
server_key_iv: "4cd287c9d9394ba98748aa1a38c15e3d"
server_key_version: "encryptionkey@0"
I20260927 00:47:56.047238 10856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-1/wal/instance:
uuid: "dcc6819721594bd18ba45d1220ca8309"
format_stamp: "Formatted at 2026-09-27 00:47:56 on dist-test-slave-86rs"
server_key: "d2e47b1783a96ba24129034f70d0d582"
server_key_iv: "4cd287c9d9394ba98748aa1a38c15e3d"
server_key_version: "encryptionkey@0"
I20260927 00:47:56.052196 10856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260927 00:47:56.055161 10871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:56.056725 10856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:47:56.056878 10856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "dcc6819721594bd18ba45d1220ca8309"
format_stamp: "Formatted at 2026-09-27 00:47:56 on dist-test-slave-86rs"
server_key: "d2e47b1783a96ba24129034f70d0d582"
server_key_iv: "4cd287c9d9394ba98748aa1a38c15e3d"
server_key_version: "encryptionkey@0"
I20260927 00:47:56.057006 10856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:56.071456 10856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:56.075125 10856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:56.075356 10856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:56.076025 10856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:56.076982 10856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:56.077032 10856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:56.077075 10856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:56.077131 10856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:56.086532 10856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32875
I20260927 00:47:56.086548 10984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32875 every 8 connection(s)
I20260927 00:47:56.087553 10856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:47:56.088641 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 10856
I20260927 00:47:56.088745 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:47:56.089046 392 external_mini_cluster.cc:1428] Setting key f8ce513da98341886b0329655afaffa8
Sep 27 00:47:56 dist-test-slave-86rs krb5kdc[10615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:47:56.105254 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:56.090077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43767 (local address 127.0.98.62:40475)
0927 00:47:56.090248 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:56.090253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:56.091205 (+ 952us) server_negotiation.cc:407] Connection header received
0927 00:47:56.092858 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:56.092861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:56.092912 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:56.092979 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:56.094690 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.095242 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.097362 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.097615 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.097778 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:56.100815 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:56.100819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:56.100821 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:56.100851 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:56.102495 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:56.103205 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:56.103209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:56.103212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:56.103347 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:56.103811 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:56.103819 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:56.103821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:56.104057 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:56.104173 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:56.104931 (+ 758us) server_negotiation.cc:299] Negotiation successful
0927 00:47:56.105073 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:47:56.106272 10987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:56.090430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40475 (local address 127.0.98.2:43767)
0927 00:47:56.091036 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:56.091081 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:56.092567 (+ 1486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:56.093198 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:56.093210 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:56.093814 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:56.094478 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:56.094496 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.095422 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.095429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:56.097181 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:56.097197 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.097838 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.097843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:56.098039 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:56.098762 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:56.098782 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:56.100567 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:56.102626 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:56.102633 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:56.102648 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:56.102953 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:56.103468 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:56.103474 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:56.103479 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:56.103668 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:56.104190 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:56.104199 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:56.104601 (+ 402us) client_negotiation.cc:769] Sending connection context
0927 00:47:56.104873 (+ 272us) client_negotiation.cc:240] Negotiation successful
0927 00:47:56.105257 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":144,"threads_started":1}
I20260927 00:47:56.107853 10985 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40475
I20260927 00:47:56.108191 10985 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:56.108903 10985 heartbeater.cc:511] Master 127.0.98.62:40475 requested a full tablet report, sending...
I20260927 00:47:56.110210 10660 ts_manager.cc:194] Registered new tserver with Master: dcc6819721594bd18ba45d1220ca8309 (127.0.98.2:32875)
I20260927 00:47:56.111145 10660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:47:56.124028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40475
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:47:56.243220 10990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:47:56.243520 10990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:47:56.243564 10990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:47:56.247710 10990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:47:56.247819 10990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:47:56.247889 10990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:47:56.253659 10990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:47:56.254808 10990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:47:56.255843 10990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:47:56.266108 10996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:56.266108 10998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:47:56.266113 10995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:47:56.266367 10990 server_base.cc:1034] running on GCE node
I20260927 00:47:56.266948 10990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:47:56.267668 10990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:47:56.268867 10990 hybrid_clock.cc:648] HybridClock initialized: now 1790470076268850 us; error 33 us; skew 500 ppm
Sep 27 00:47:56 dist-test-slave-86rs krb5kdc[10615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:47:56.272039 10990 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:47:56.273150 10990 webserver.cc:460] Webserver started at http://127.0.98.3:37913/ using document root <none> and password file <none>
I20260927 00:47:56.273747 10990 fs_manager.cc:359] Metadata directory not provided
I20260927 00:47:56.273808 10990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:47:56.273981 10990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:47:56.276142 10990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-2/data/instance:
uuid: "a1b82a35f4dd40668ecd42ca5f934aba"
format_stamp: "Formatted at 2026-09-27 00:47:56 on dist-test-slave-86rs"
server_key: "e792463e7fabc3cc531d40bb8c6221a5"
server_key_iv: "f8a688dd5111a68f9b525541a88174e9"
server_key_version: "encryptionkey@0"
I20260927 00:47:56.276814 10990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a1b82a35f4dd40668ecd42ca5f934aba"
format_stamp: "Formatted at 2026-09-27 00:47:56 on dist-test-slave-86rs"
server_key: "e792463e7fabc3cc531d40bb8c6221a5"
server_key_iv: "f8a688dd5111a68f9b525541a88174e9"
server_key_version: "encryptionkey@0"
I20260927 00:47:56.280973 10990 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:47:56.283592 11005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:56.285034 10990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:47:56.285189 10990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "a1b82a35f4dd40668ecd42ca5f934aba"
format_stamp: "Formatted at 2026-09-27 00:47:56 on dist-test-slave-86rs"
server_key: "e792463e7fabc3cc531d40bb8c6221a5"
server_key_iv: "f8a688dd5111a68f9b525541a88174e9"
server_key_version: "encryptionkey@0"
I20260927 00:47:56.285320 10990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:47:56.308275 10990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:47:56.311968 10990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:47:56.312189 10990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:47:56.312875 10990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:47:56.313812 10990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:47:56.313886 10990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:56.313958 10990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:47:56.314009 10990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:47:56.323712 10990 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40801
I20260927 00:47:56.323727 11118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40801 every 8 connection(s)
I20260927 00:47:56.324800 10990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:47:56.331784 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 10990
I20260927 00:47:56.331930 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:47:56.332274 392 external_mini_cluster.cc:1428] Setting key cdb86c145581e9e679376a91a6480b8f
Sep 27 00:47:56 dist-test-slave-86rs krb5kdc[10615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:47:56.340405 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:56.326825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42357 (local address 127.0.98.62:40475)
0927 00:47:56.326992 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:56.326996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:56.327953 (+ 957us) server_negotiation.cc:407] Connection header received
0927 00:47:56.329351 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:56.329354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:56.329407 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:56.329491 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:56.330905 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.331474 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.333245 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.333583 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.333728 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:56.336585 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:56.336589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:56.336591 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:56.336645 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:56.338088 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:56.338680 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:56.338683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:56.338685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:56.338773 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:56.339199 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:56.339203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:56.339205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:56.339413 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:56.339522 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:56.340067 (+ 545us) server_negotiation.cc:299] Negotiation successful
0927 00:47:56.340200 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 00:47:56.341328 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:56.327217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40475 (local address 127.0.98.3:42357)
0927 00:47:56.327714 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:56.327763 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:56.329082 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:56.329670 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:56.329677 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:56.330093 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:47:56.330737 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:56.330752 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.331635 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.331640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:56.333071 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:56.333081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.333767 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.333772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:56.333992 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:56.334719 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:47:56.334737 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:47:56.336395 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:47:56.338236 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:56.338243 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:56.338265 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:56.338552 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:56.338872 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:47:56.338875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:47:56.338877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:47:56.339044 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:47:56.339521 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:47:56.339531 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:47:56.339816 (+ 285us) client_negotiation.cc:769] Sending connection context
0927 00:47:56.340014 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 00:47:56.340430 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":113,"threads_started":1}
I20260927 00:47:56.342641 11119 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40475
I20260927 00:47:56.342967 11119 heartbeater.cc:464] Registering TS with master...
I20260927 00:47:56.343539 11119 heartbeater.cc:511] Master 127.0.98.62:40475 requested a full tablet report, sending...
I20260927 00:47:56.344766 10660 ts_manager.cc:194] Registered new tserver with Master: a1b82a35f4dd40668ecd42ca5f934aba (127.0.98.3:40801)
I20260927 00:47:56.345533 10660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42357
I20260927 00:47:56.346503 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:47:56.356657 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:56.348751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56274 (local address 127.0.98.62:40475)
0927 00:47:56.348865 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:56.348869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:56.349031 (+ 162us) server_negotiation.cc:407] Connection header received
0927 00:47:56.349165 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:56.349168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:56.349218 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:56.349307 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:47:56.350078 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.350598 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.351495 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.351720 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.351830 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:56.353348 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:47:56.353359 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:47:56.353361 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:47:56.353385 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:47:56.354770 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:56.355262 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:56.355266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:56.355268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:56.355357 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:47:56.355646 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:47:56.355650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:47:56.355651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:47:56.355891 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:47:56.356048 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:56.356339 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 00:47:56.356463 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 00:47:56.359741 10660 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:47:56.363072 10660 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:47:56.378172 11133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:56.372069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37543 (local address 127.0.0.1:51934)
0927 00:47:56.372909 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:56.372961 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:56.373103 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:56.373845 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:56.373850 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:56.373874 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:56.374040 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:56.374046 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.375525 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.375529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:56.376526 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:56.376534 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.377584 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.377588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:56.377692 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:56.377730 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 00:47:56.377872 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 00:47:56.378005 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"spinlock_wait_cycles":7424,"thread_start_us":110,"threads_started":1}
I20260927 00:47:56.378168 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:56.372218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51934 (local address 127.0.98.1:37543)
0927 00:47:56.372658 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:56.372666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:56.373049 (+ 383us) server_negotiation.cc:407] Connection header received
0927 00:47:56.373284 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:56.373290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:56.373516 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:56.373643 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:56.374178 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.375381 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.376736 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.377459 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.377586 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:56.377753 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:56.377904 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 00:47:56.378004 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":101,"threads_started":1}
I20260927 00:47:56.378459 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:56.372079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40801 (local address 127.0.0.1:59298)
0927 00:47:56.372930 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:56.372961 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:56.373091 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:56.374032 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:56.374035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:56.374050 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:56.374140 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:56.374148 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.375657 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.375661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:56.376921 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:56.376927 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.378033 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.378037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:56.378149 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:56.378170 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:47:56.378222 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 00:47:56.378301 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":49,"thread_start_us":112,"threads_started":1}
I20260927 00:47:56.378641 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:56.372417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32875 (local address 127.0.0.1:43126)
0927 00:47:56.373299 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:56.373310 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:56.373374 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:56.373873 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:56.373876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:56.373893 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:56.374048 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:56.374056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.375201 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.375204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:56.376899 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:56.376909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.378033 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.378037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:56.378351 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:56.378371 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 00:47:56.378438 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 00:47:56.378514 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":306,"thread_start_us":81,"threads_started":1}
I20260927 00:47:56.379245 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:56.373126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59298 (local address 127.0.98.3:40801)
0927 00:47:56.373603 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:56.373614 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:56.373635 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:47:56.373716 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:56.373720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:56.373891 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:56.374279 (+ 388us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:56.374437 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.375522 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.377047 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.377888 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.378564 (+ 676us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:56.378984 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:56.379035 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 00:47:56.379134 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":149,"threads_started":1}
I20260927 00:47:56.378736 11134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:56.372958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43126 (local address 127.0.98.2:32875)
0927 00:47:56.373402 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:56.373414 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:56.373438 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:47:56.373517 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:56.373524 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:56.373714 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:56.374214 (+ 500us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:56.374341 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.375049 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.377048 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.377888 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.378080 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:56.378235 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:56.378478 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 00:47:56.378574 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":121,"threads_started":1}
I20260927 00:47:56.381822 11053 tablet_service.cc:1467] Processing CreateTablet for tablet 28faef6a0ac94833a09acf9090c67223 (DEFAULT_TABLE table=test-table [id=65d7bf050d0a44a5b0741249acfbe9c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:56.381844 10785 tablet_service.cc:1467] Processing CreateTablet for tablet 28faef6a0ac94833a09acf9090c67223 (DEFAULT_TABLE table=test-table [id=65d7bf050d0a44a5b0741249acfbe9c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:56.382297 10919 tablet_service.cc:1467] Processing CreateTablet for tablet 28faef6a0ac94833a09acf9090c67223 (DEFAULT_TABLE table=test-table [id=65d7bf050d0a44a5b0741249acfbe9c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:47:56.382941 11053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28faef6a0ac94833a09acf9090c67223. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:56.382937 10785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28faef6a0ac94833a09acf9090c67223. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:56.383239 10919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28faef6a0ac94833a09acf9090c67223. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:47:56.389684 11136 tablet_bootstrap.cc:492] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba: Bootstrap starting.
I20260927 00:47:56.391516 11138 tablet_bootstrap.cc:492] T 28faef6a0ac94833a09acf9090c67223 P dcc6819721594bd18ba45d1220ca8309: Bootstrap starting.
I20260927 00:47:56.392213 11136 tablet_bootstrap.cc:654] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:56.392601 11137 tablet_bootstrap.cc:492] T 28faef6a0ac94833a09acf9090c67223 P d6b096bba43c463194042a895634d6db: Bootstrap starting.
I20260927 00:47:56.393565 11136 log.cc:824] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:56.394165 11138 tablet_bootstrap.cc:654] T 28faef6a0ac94833a09acf9090c67223 P dcc6819721594bd18ba45d1220ca8309: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:56.394902 11137 tablet_bootstrap.cc:654] T 28faef6a0ac94833a09acf9090c67223 P d6b096bba43c463194042a895634d6db: Neither blocks nor log segments found. Creating new log.
I20260927 00:47:56.395048 11138 log.cc:824] T 28faef6a0ac94833a09acf9090c67223 P dcc6819721594bd18ba45d1220ca8309: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:56.395674 11137 log.cc:824] T 28faef6a0ac94833a09acf9090c67223 P d6b096bba43c463194042a895634d6db: Log is configured to *not* fsync() on all Append() calls
I20260927 00:47:56.395900 11136 tablet_bootstrap.cc:492] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba: No bootstrap required, opened a new log
I20260927 00:47:56.396145 11136 ts_tablet_manager.cc:1397] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 00:47:56.397244 11138 tablet_bootstrap.cc:492] T 28faef6a0ac94833a09acf9090c67223 P dcc6819721594bd18ba45d1220ca8309: No bootstrap required, opened a new log
I20260927 00:47:56.397444 11138 ts_tablet_manager.cc:1397] T 28faef6a0ac94833a09acf9090c67223 P dcc6819721594bd18ba45d1220ca8309: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 00:47:56.397729 11137 tablet_bootstrap.cc:492] T 28faef6a0ac94833a09acf9090c67223 P d6b096bba43c463194042a895634d6db: No bootstrap required, opened a new log
I20260927 00:47:56.397905 11137 ts_tablet_manager.cc:1397] T 28faef6a0ac94833a09acf9090c67223 P d6b096bba43c463194042a895634d6db: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:47:56.399567 11136 raft_consensus.cc:348] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b096bba43c463194042a895634d6db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37543 } } peers { permanent_uuid: "a1b82a35f4dd40668ecd42ca5f934aba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40801 } } peers { permanent_uuid: "dcc6819721594bd18ba45d1220ca8309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32875 } }
I20260927 00:47:56.400017 11136 raft_consensus.cc:374] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:56.400131 11136 raft_consensus.cc:729] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1b82a35f4dd40668ecd42ca5f934aba, State: Initialized, Role: FOLLOWER
I20260927 00:47:56.400730 11136 consensus_queue.cc:260] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b096bba43c463194042a895634d6db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37543 } } peers { permanent_uuid: "a1b82a35f4dd40668ecd42ca5f934aba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40801 } } peers { permanent_uuid: "dcc6819721594bd18ba45d1220ca8309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32875 } }
I20260927 00:47:56.401437 11119 heartbeater.cc:503] Master 127.0.98.62:40475 was elected leader, sending a full tablet report...
I20260927 00:47:56.401773 11138 raft_consensus.cc:348] T 28faef6a0ac94833a09acf9090c67223 P dcc6819721594bd18ba45d1220ca8309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b096bba43c463194042a895634d6db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37543 } } peers { permanent_uuid: "a1b82a35f4dd40668ecd42ca5f934aba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40801 } } peers { permanent_uuid: "dcc6819721594bd18ba45d1220ca8309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32875 } }
I20260927 00:47:56.402068 11138 raft_consensus.cc:374] T 28faef6a0ac94833a09acf9090c67223 P dcc6819721594bd18ba45d1220ca8309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:56.402174 11138 raft_consensus.cc:729] T 28faef6a0ac94833a09acf9090c67223 P dcc6819721594bd18ba45d1220ca8309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc6819721594bd18ba45d1220ca8309, State: Initialized, Role: FOLLOWER
I20260927 00:47:56.402484 11137 raft_consensus.cc:348] T 28faef6a0ac94833a09acf9090c67223 P d6b096bba43c463194042a895634d6db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b096bba43c463194042a895634d6db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37543 } } peers { permanent_uuid: "a1b82a35f4dd40668ecd42ca5f934aba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40801 } } peers { permanent_uuid: "dcc6819721594bd18ba45d1220ca8309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32875 } }
I20260927 00:47:56.402679 11138 consensus_queue.cc:260] T 28faef6a0ac94833a09acf9090c67223 P dcc6819721594bd18ba45d1220ca8309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b096bba43c463194042a895634d6db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37543 } } peers { permanent_uuid: "a1b82a35f4dd40668ecd42ca5f934aba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40801 } } peers { permanent_uuid: "dcc6819721594bd18ba45d1220ca8309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32875 } }
I20260927 00:47:56.402783 11137 raft_consensus.cc:374] T 28faef6a0ac94833a09acf9090c67223 P d6b096bba43c463194042a895634d6db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:47:56.402858 11137 raft_consensus.cc:729] T 28faef6a0ac94833a09acf9090c67223 P d6b096bba43c463194042a895634d6db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b096bba43c463194042a895634d6db, State: Initialized, Role: FOLLOWER
I20260927 00:47:56.402916 11136 ts_tablet_manager.cc:1428] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 00:47:56.403354 11137 consensus_queue.cc:260] T 28faef6a0ac94833a09acf9090c67223 P d6b096bba43c463194042a895634d6db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b096bba43c463194042a895634d6db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37543 } } peers { permanent_uuid: "a1b82a35f4dd40668ecd42ca5f934aba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40801 } } peers { permanent_uuid: "dcc6819721594bd18ba45d1220ca8309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32875 } }
I20260927 00:47:56.403756 11138 ts_tablet_manager.cc:1428] T 28faef6a0ac94833a09acf9090c67223 P dcc6819721594bd18ba45d1220ca8309: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:47:56.404043 10851 heartbeater.cc:503] Master 127.0.98.62:40475 was elected leader, sending a full tablet report...
I20260927 00:47:56.404119 10985 heartbeater.cc:503] Master 127.0.98.62:40475 was elected leader, sending a full tablet report...
I20260927 00:47:56.404291 11137 ts_tablet_manager.cc:1428] T 28faef6a0ac94833a09acf9090c67223 P d6b096bba43c463194042a895634d6db: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260927 00:47:56.556356 11142 raft_consensus.cc:482] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:47:56.556596 11142 raft_consensus.cc:504] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b096bba43c463194042a895634d6db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37543 } } peers { permanent_uuid: "a1b82a35f4dd40668ecd42ca5f934aba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40801 } } peers { permanent_uuid: "dcc6819721594bd18ba45d1220ca8309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32875 } }
I20260927 00:47:56.557875 11142 leader_election.cc:288] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6b096bba43c463194042a895634d6db (127.0.98.1:37543), dcc6819721594bd18ba45d1220ca8309 (127.0.98.2:32875)
I20260927 00:47:56.563277 11134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:56.558299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45231 (local address 127.0.98.2:32875)
0927 00:47:56.558449 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:56.558456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:56.558475 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:47:56.558562 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:56.558568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:56.558625 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:56.558695 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:56.559032 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.559880 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.561825 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.562454 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.562584 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:56.562656 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:56.563037 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 00:47:56.563128 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 00:47:56.563277 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:56.558311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39173 (local address 127.0.98.1:37543)
0927 00:47:56.558430 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:56.558434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:56.558835 (+ 401us) server_negotiation.cc:407] Connection header received
0927 00:47:56.558908 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:56.558910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:56.558951 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:56.559048 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:47:56.559314 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.559998 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.561825 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.562473 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.562582 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:56.562657 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:56.563036 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 00:47:56.563145 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:47:56.563337 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:56.558174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32875 (local address 127.0.98.3:45231)
0927 00:47:56.558325 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:56.558343 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:56.558459 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:56.558734 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:56.558739 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:56.558764 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:56.558904 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:56.558912 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.560067 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.560074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:56.561670 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:56.561679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.562625 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.562630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:56.562784 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:56.562886 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 00:47:56.562979 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 00:47:56.563123 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":242}
I20260927 00:47:56.563344 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:56.558174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37543 (local address 127.0.98.3:39173)
0927 00:47:56.558726 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:56.558740 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:47:56.558830 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:47:56.559074 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:47:56.559076 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:47:56.559091 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:47:56.559182 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:56.559186 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.560100 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.560103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:56.561667 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:47:56.561679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.562625 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.562630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:47:56.562851 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:56.562905 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 00:47:56.562978 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 00:47:56.563124 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":327,"thread_start_us":128,"threads_started":1}
I20260927 00:47:56.564427 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28faef6a0ac94833a09acf9090c67223" candidate_uuid: "a1b82a35f4dd40668ecd42ca5f934aba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc6819721594bd18ba45d1220ca8309" is_pre_election: true
I20260927 00:47:56.564481 10805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28faef6a0ac94833a09acf9090c67223" candidate_uuid: "a1b82a35f4dd40668ecd42ca5f934aba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b096bba43c463194042a895634d6db" is_pre_election: true
I20260927 00:47:56.564795 10939 raft_consensus.cc:2454] T 28faef6a0ac94833a09acf9090c67223 P dcc6819721594bd18ba45d1220ca8309 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1b82a35f4dd40668ecd42ca5f934aba in term 0.
I20260927 00:47:56.564834 10805 raft_consensus.cc:2454] T 28faef6a0ac94833a09acf9090c67223 P d6b096bba43c463194042a895634d6db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1b82a35f4dd40668ecd42ca5f934aba in term 0.
I20260927 00:47:56.565311 11007 leader_election.cc:302] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1b82a35f4dd40668ecd42ca5f934aba, dcc6819721594bd18ba45d1220ca8309; no voters:
I20260927 00:47:56.565687 11142 raft_consensus.cc:2781] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:47:56.565815 11142 raft_consensus.cc:482] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:47:56.565876 11142 raft_consensus.cc:3037] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:56.567101 11142 raft_consensus.cc:504] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b096bba43c463194042a895634d6db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37543 } } peers { permanent_uuid: "a1b82a35f4dd40668ecd42ca5f934aba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40801 } } peers { permanent_uuid: "dcc6819721594bd18ba45d1220ca8309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32875 } }
I20260927 00:47:56.567495 11142 leader_election.cc:288] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba [CANDIDATE]: Term 1 election: Requested vote from peers d6b096bba43c463194042a895634d6db (127.0.98.1:37543), dcc6819721594bd18ba45d1220ca8309 (127.0.98.2:32875)
I20260927 00:47:56.567924 10805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28faef6a0ac94833a09acf9090c67223" candidate_uuid: "a1b82a35f4dd40668ecd42ca5f934aba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b096bba43c463194042a895634d6db"
I20260927 00:47:56.568029 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28faef6a0ac94833a09acf9090c67223" candidate_uuid: "a1b82a35f4dd40668ecd42ca5f934aba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc6819721594bd18ba45d1220ca8309"
I20260927 00:47:56.568087 10805 raft_consensus.cc:3037] T 28faef6a0ac94833a09acf9090c67223 P d6b096bba43c463194042a895634d6db [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:56.568219 10939 raft_consensus.cc:3037] T 28faef6a0ac94833a09acf9090c67223 P dcc6819721594bd18ba45d1220ca8309 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:47:56.569265 10805 raft_consensus.cc:2454] T 28faef6a0ac94833a09acf9090c67223 P d6b096bba43c463194042a895634d6db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1b82a35f4dd40668ecd42ca5f934aba in term 1.
I20260927 00:47:56.569301 10939 raft_consensus.cc:2454] T 28faef6a0ac94833a09acf9090c67223 P dcc6819721594bd18ba45d1220ca8309 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1b82a35f4dd40668ecd42ca5f934aba in term 1.
I20260927 00:47:56.569607 11008 leader_election.cc:302] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1b82a35f4dd40668ecd42ca5f934aba, d6b096bba43c463194042a895634d6db; no voters:
I20260927 00:47:56.569819 11142 raft_consensus.cc:2781] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:47:56.570098 11142 raft_consensus.cc:686] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba [term 1 LEADER]: Becoming Leader. State: Replica: a1b82a35f4dd40668ecd42ca5f934aba, State: Running, Role: LEADER
I20260927 00:47:56.570487 11142 consensus_queue.cc:237] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b096bba43c463194042a895634d6db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37543 } } peers { permanent_uuid: "a1b82a35f4dd40668ecd42ca5f934aba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40801 } } peers { permanent_uuid: "dcc6819721594bd18ba45d1220ca8309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32875 } }
I20260927 00:47:56.573943 10659 catalog_manager.cc:5495] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba reported cstate change: term changed from 0 to 1, leader changed from <none> to a1b82a35f4dd40668ecd42ca5f934aba (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a1b82a35f4dd40668ecd42ca5f934aba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b096bba43c463194042a895634d6db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1b82a35f4dd40668ecd42ca5f934aba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcc6819721594bd18ba45d1220ca8309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32875 } health_report { overall_health: UNKNOWN } } }
W20260927 00:47:56.575790 11120 tablet.cc:2369] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:47:56.588970 10986 tablet.cc:2369] T 28faef6a0ac94833a09acf9090c67223 P dcc6819721594bd18ba45d1220ca8309: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:47:56.605360 10852 tablet.cc:2369] T 28faef6a0ac94833a09acf9090c67223 P d6b096bba43c463194042a895634d6db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:47:56.622009 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:56.617910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59312 (local address 127.0.98.3:40801)
0927 00:47:56.618054 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:47:56.618057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:47:56.618072 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:47:56.618216 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:47:56.618219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:47:56.618289 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:47:56.618385 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:47:56.618604 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.619108 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.620468 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:47:56.620690 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:47:56.620794 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:47:56.621108 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:47:56.621552 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:47:56.621669 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 00:47:56.621813 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 00:47:56.621868 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 00:47:56.628307 10939 raft_consensus.cc:1261] T 28faef6a0ac94833a09acf9090c67223 P dcc6819721594bd18ba45d1220ca8309 [term 1 FOLLOWER]: Refusing update from remote peer a1b82a35f4dd40668ecd42ca5f934aba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:47:56.628337 10805 raft_consensus.cc:1261] T 28faef6a0ac94833a09acf9090c67223 P d6b096bba43c463194042a895634d6db [term 1 FOLLOWER]: Refusing update from remote peer a1b82a35f4dd40668ecd42ca5f934aba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:47:56.629153 11142 consensus_queue.cc:1035] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b096bba43c463194042a895634d6db" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:47:56.629371 11146 consensus_queue.cc:1035] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcc6819721594bd18ba45d1220ca8309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:47:56.637929 11151 mvcc.cc:205] Tried to move back new op lower bound from 7333765433861562368 to 7333765433635942400. Current Snapshot: MvccSnapshot[applied={T|T < 7333765433861562368}]
I20260927 00:47:56.639120 11150 mvcc.cc:205] Tried to move back new op lower bound from 7333765433861562368 to 7333765433635942400. Current Snapshot: MvccSnapshot[applied={T|T < 7333765433861562368}]
I20260927 00:47:56.649097 10658 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:47:56.649273 10658 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:47:56.652215 10658 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 86b4ffbb571a4973aa3738c7af8a6d78: Sending DeleteTablet for 3 replicas of tablet 28faef6a0ac94833a09acf9090c67223
I20260927 00:47:56.653239 11053 tablet_service.cc:1514] Processing DeleteTablet for tablet 28faef6a0ac94833a09acf9090c67223 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59298
I20260927 00:47:56.653415 10785 tablet_service.cc:1514] Processing DeleteTablet for tablet 28faef6a0ac94833a09acf9090c67223 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51934
I20260927 00:47:56.653400 10919 tablet_service.cc:1514] Processing DeleteTablet for tablet 28faef6a0ac94833a09acf9090c67223 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:47:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43126
I20260927 00:47:56.654001 11158 tablet_replica.cc:330] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba: stopping tablet replica
I20260927 00:47:56.654392 11158 raft_consensus.cc:2229] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:47:56.654641 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 10722
I20260927 00:47:56.656297 11158 raft_consensus.cc:2258] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:47:56.658259 11158 ts_tablet_manager.cc:1905] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:47:56.661300 11158 ts_tablet_manager.cc:1918] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:47:56.661490 11158 log.cc:1196] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790470006085796-392-0/minicluster-data/ts-2/wal/wals/28faef6a0ac94833a09acf9090c67223
I20260927 00:47:56.661862 11158 ts_tablet_manager.cc:1939] T 28faef6a0ac94833a09acf9090c67223 P a1b82a35f4dd40668ecd42ca5f934aba: Deleting consensus metadata
I20260927 00:47:56.663326 10644 catalog_manager.cc:4841] TS a1b82a35f4dd40668ecd42ca5f934aba (127.0.98.3:40801): tablet 28faef6a0ac94833a09acf9090c67223 (table test-table [id=65d7bf050d0a44a5b0741249acfbe9c6]) successfully deleted
I20260927 00:47:56.664496 11160 tablet_replica.cc:330] T 28faef6a0ac94833a09acf9090c67223 P dcc6819721594bd18ba45d1220ca8309: stopping tablet replica
I20260927 00:47:56.664906 11160 raft_consensus.cc:2229] T 28faef6a0ac94833a09acf9090c67223 P dcc6819721594bd18ba45d1220ca8309 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:47:56.665323 11160 raft_consensus.cc:2258] T 28faef6a0ac94833a09acf9090c67223 P dcc6819721594bd18ba45d1220ca8309 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 00:47:56.665331 10646 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37543 (error 108)
I20260927 00:47:56.666057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 10856
I20260927 00:47:56.668995 11133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:56.668625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37543 (local address 127.0.0.1:51944)
0927 00:47:56.668797 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:56.668891 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260927 00:47:56.669215 10646 catalog_manager.cc:4584] TS d6b096bba43c463194042a895634d6db (127.0.98.1:37543): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 28faef6a0ac94833a09acf9090c67223: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37543: connect: Connection refused (error 111)
I20260927 00:47:56.673141 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 10990
I20260927 00:47:56.674101 11133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:47:56.673324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32875 (local address 127.0.0.1:43134)
0927 00:47:56.673922 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0927 00:47:56.673991 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491}
I20260927 00:47:56.679147 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 10628
2026-09-27T00:47:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2548 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:47:56 dist-test-slave-86rs krb5kdc[11164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:47:56 dist-test-slave-86rs krb5kdc[11164](info): set up 2 sockets
Sep 27 00:47:56 dist-test-slave-86rs krb5kdc[11164](info): commencing operation
krb5kdc: starting...
W20260927 00:47:57.729127 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:47:57 dist-test-slave-86rs krb5kdc[11164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:47:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:47:57Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 00:47:57.911005 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskMaietF/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/postgres -l logfile start
2026-09-27 00:48:00.662 UTC [11194] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 00:48:00.662 UTC [11194] LOG: listening on IPv4 address "127.0.98.20", port 49845
2026-09-27 00:48:00.666 UTC [11194] LOG: listening on Unix socket "/tmp/.s.PGSQL.49845"
2026-09-27 00:48:00.698 UTC [11197] LOG: database system was shut down at 2026-09-27 00:47:58 UTC
2026-09-27 00:48:00.702 UTC [11194] LOG: database system is ready to accept connections
I20260927 00:48:01.661746 392 mini_postgres.cc:96] Postgres bound to 49845
2026-09-27 00:48:01.668 UTC [11205] FATAL: database "slave" does not exist
127.0.98.20:49845 - accepting connections
I20260927 00:48:01.668978 392 mini_ranger.cc:161] Starting Ranger...
I20260927 00:48:01.684921 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 00:48:02.270133 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 00:48:02.270234 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ranger-admin
2026-09-27 00:48:02,596 [I] DB FLAVOR :POSTGRES
2026-09-27 00:48:02,597 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 00:48:02,597 [I] Checking connection..
2026-09-27 00:48:02,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 00:48:02,892 [I] Checking connection passed.
2026-09-27 00:48:02,893 [I] --------- Verifying version history table ---------
2026-09-27 00:48:02,893 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:48:03,191 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 00:48:03,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:48:03,464 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 00:48:03,465 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 00:48:03,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49845/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 00:48:03.744 UTC [11269] WARNING: there is no transaction in progress
2026-09-27 00:48:03,752 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 00:48:03,752 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:48:04,050 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 00:48:04,050 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 00:48:04,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 00:48:04,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 00:48:04,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-86rs.c.gcp-upstream.internal','N') ;"
2026-09-27 00:48:04,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 00:48:05,161 [I] Table x_portal_user does not exist in database ranger
2026-09-27 00:48:05,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 00:48:05,464 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 00:48:05,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 00:48:05,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 00:48:06,020 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 00:48:06,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49845/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 00:48:07.150 UTC [11377] WARNING: there is no transaction in progress
2026-09-27 00:48:07.169 UTC [11377] WARNING: there is no transaction in progress
2026-09-27 00:48:07.189 UTC [11377] WARNING: there is no transaction in progress
2026-09-27 00:48:07.208 UTC [11377] WARNING: there is no transaction in progress
2026-09-27 00:48:07.228 UTC [11377] WARNING: there is no transaction in progress
2026-09-27 00:48:07.248 UTC [11377] WARNING: there is no transaction in progress
2026-09-27 00:48:07.308 UTC [11377] WARNING: there is no transaction in progress
2026-09-27 00:48:07.316 UTC [11377] WARNING: there is no transaction in progress
2026-09-27 00:48:07.333 UTC [11377] WARNING: there is no transaction in progress
2026-09-27 00:48:07.346 UTC [11377] WARNING: there is no transaction in progress
2026-09-27 00:48:07.358 UTC [11377] WARNING: there is no transaction in progress
2026-09-27 00:48:07.374 UTC [11377] WARNING: there is no transaction in progress
2026-09-27 00:48:07.385 UTC [11377] WARNING: there is no transaction in progress
2026-09-27 00:48:07.392 UTC [11377] WARNING: there is no transaction in progress
2026-09-27 00:48:07.401 UTC [11377] WARNING: there is no transaction in progress
2026-09-27 00:48:07.409 UTC [11377] WARNING: there is no transaction in progress
2026-09-27 00:48:08,171 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 00:48:08,172 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 00:48:08,434 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 00:48:08,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 00:48:08,711 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 00:48:08,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 00:48:08,982 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 00:48:08,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 00:48:09,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 00:48:09,534 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-86rs.c.gcp-upstream.internal';"
2026-09-27 00:48:09,816 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 00:48:09,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49845/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 00:48:10,070 [I] DB_PATCHES have already been applied
I20260927 00:48:10.076735 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskMaietF/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 00:48:10.076848 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 00:48:10.081489 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:60463
Sep 27, 2026 12:48:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:48:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:48:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 12:48:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 12:48:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 12:48:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 12:48:11 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 00:48:11.231226 392 mini_ranger.cc:160] Time spent starting Ranger: real 9.562s user 0.004s sys 0.009s
Sep 27, 2026 12:48:32 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 12:48:32 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 12:48:32 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 12:48:32 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 12:48:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 12:48:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 12:48:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 12:48:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 12:48:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 12:48:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 12:48:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 12:48:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 12:48:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 12:48:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 12:48:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 12:48:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 12:48:32 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 12:48:33 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 00:48:34.397817 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 00:48:34 dist-test-slave-86rs krb5kdc[11164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470114, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 00:48:34.486853 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 00:48:34.496982 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 00:48:35.468147 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 00:48:35.468290 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ranger-kms
I20260927 00:48:35.468432 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:49845
2026-09-27 00:48:35,661 [I] DB FLAVOR :POSTGRES
2026-09-27 00:48:35,662 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 00:48:35,662 [I] Checking connection
2026-09-27 00:48:35,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49845/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 00:48:35,929 [I] connection success
2026-09-27 00:48:35,929 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 00:48:35,929 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 00:48:35,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49845/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 00:48:36,190 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 00:48:36,191 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 00:48:36,191 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 00:48:36,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49845/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 00:48:36,483 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 00:48:37.008575 392 mini_ranger_kms.cc:310] Created kms service
I20260927 00:48:37.201262 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 00:48:37.246572 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 00:48:37 dist-test-slave-86rs krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470114, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:48:37.646871 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 00:48:37.647212 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 00:48:37.647251 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 00:48:37.650990 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11558
I20260927 00:48:37.651141 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:38399
Sep 27, 2026 12:48:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:48:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:48:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 12:48:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 12:48:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 12:48:39 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 00:48:40.003407 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.517s user 0.005s sys 0.172s
I20260927 00:48:40.003540 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 00:48:40.003695 392 mini_ranger_kms.cc:401] 127.0.98.20:38399/kms/v1/keys
Sep 27 00:48:40 dist-test-slave-86rs krb5kdc[11164](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790470120, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 00:48:40 dist-test-slave-86rs krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470120, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 00:48:40 dist-test-slave-86rs krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470120, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 00:48:43 dist-test-slave-86rs krb5kdc[11164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:48:43.328759 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42309
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42309
--ranger_config_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38399/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:43.449189 11608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:43.449525 11608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:43.449611 11608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:43.453742 11608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:48:43.453814 11608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:43.453832 11608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:43.453850 11608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:48:43.453864 11608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:48:43.459558 11608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:38399/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42309
--ranger_config_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:48:43.460743 11608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:43.461826 11608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:43.471133 11616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:43.471186 11608 server_base.cc:1034] running on GCE node
W20260927 00:48:43.471149 11614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:43.471133 11613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:43.471923 11608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:43.473194 11608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:43.474419 11608 hybrid_clock.cc:648] HybridClock initialized: now 1790470123474400 us; error 30 us; skew 500 ppm
Sep 27 00:48:43 dist-test-slave-86rs krb5kdc[11164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:43.480892 11608 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:48:43.481922 11608 webserver.cc:460] Webserver started at http://127.0.98.62:38779/ using document root <none> and password file <none>
I20260927 00:48:43.482530 11608 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:43.482618 11608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:43.482882 11608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:48:43 dist-test-slave-86rs krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:48:43.530138 11608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "946be0b9236e4f04909359384ffe4152"
format_stamp: "Formatted at 2026-09-27 00:48:43 on dist-test-slave-86rs"
server_key: "1c9e136cde3f68f0dbea39044206519c"
server_key_iv: "699293db72097b0874d0de94a8ac1f46"
server_key_version: "kuduclusterkey@0"
I20260927 00:48:43.530817 11608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "946be0b9236e4f04909359384ffe4152"
format_stamp: "Formatted at 2026-09-27 00:48:43 on dist-test-slave-86rs"
server_key: "1c9e136cde3f68f0dbea39044206519c"
server_key_iv: "699293db72097b0874d0de94a8ac1f46"
server_key_version: "kuduclusterkey@0"
I20260927 00:48:43.535110 11608 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 00:48:43.554409 11623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:43.556182 11608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:48:43.556414 11608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "946be0b9236e4f04909359384ffe4152"
format_stamp: "Formatted at 2026-09-27 00:48:43 on dist-test-slave-86rs"
server_key: "1c9e136cde3f68f0dbea39044206519c"
server_key_iv: "699293db72097b0874d0de94a8ac1f46"
server_key_version: "kuduclusterkey@0"
I20260927 00:48:43.556574 11608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:43.576668 11608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:43.579886 11608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:43.580097 11608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:43.587657 11608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42309
I20260927 00:48:43.587697 11675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42309 every 8 connection(s)
I20260927 00:48:43.588788 11608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:48:43.592435 11676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:43.597875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 11608
I20260927 00:48:43.598013 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:48:43.598307 392 external_mini_cluster.cc:1428] Setting key 36b43946f41542daf1c0132e682c7bb6
I20260927 00:48:43.598819 11676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 946be0b9236e4f04909359384ffe4152: Bootstrap starting.
I20260927 00:48:43.601511 11676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 946be0b9236e4f04909359384ffe4152: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:43.602334 11676 log.cc:824] T 00000000000000000000000000000000 P 946be0b9236e4f04909359384ffe4152: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:43.604354 11676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 946be0b9236e4f04909359384ffe4152: No bootstrap required, opened a new log
Sep 27 00:48:43 dist-test-slave-86rs krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:48:43.607834 11676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 946be0b9236e4f04909359384ffe4152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946be0b9236e4f04909359384ffe4152" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42309 } }
I20260927 00:48:43.608107 11676 raft_consensus.cc:374] T 00000000000000000000000000000000 P 946be0b9236e4f04909359384ffe4152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:43.608189 11676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 946be0b9236e4f04909359384ffe4152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 946be0b9236e4f04909359384ffe4152, State: Initialized, Role: FOLLOWER
I20260927 00:48:43.608752 11676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 946be0b9236e4f04909359384ffe4152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946be0b9236e4f04909359384ffe4152" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42309 } }
I20260927 00:48:43.608919 11676 raft_consensus.cc:388] T 00000000000000000000000000000000 P 946be0b9236e4f04909359384ffe4152 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:48:43.608992 11676 raft_consensus.cc:482] T 00000000000000000000000000000000 P 946be0b9236e4f04909359384ffe4152 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:48:43.609130 11676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 946be0b9236e4f04909359384ffe4152 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:43.610162 11676 raft_consensus.cc:504] T 00000000000000000000000000000000 P 946be0b9236e4f04909359384ffe4152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946be0b9236e4f04909359384ffe4152" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42309 } }
I20260927 00:48:43.610526 11676 leader_election.cc:302] T 00000000000000000000000000000000 P 946be0b9236e4f04909359384ffe4152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 946be0b9236e4f04909359384ffe4152; no voters:
I20260927 00:48:43.610847 11676 leader_election.cc:288] T 00000000000000000000000000000000 P 946be0b9236e4f04909359384ffe4152 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:48:43.610962 11681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 946be0b9236e4f04909359384ffe4152 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:43.611184 11681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 946be0b9236e4f04909359384ffe4152 [term 1 LEADER]: Becoming Leader. State: Replica: 946be0b9236e4f04909359384ffe4152, State: Running, Role: LEADER
I20260927 00:48:43.612180 11681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 946be0b9236e4f04909359384ffe4152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946be0b9236e4f04909359384ffe4152" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42309 } }
I20260927 00:48:43.612275 11676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 946be0b9236e4f04909359384ffe4152 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:48:43.613138 11679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:43.599880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43570 (local address 127.0.98.62:42309)
0927 00:48:43.600420 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:43.600436 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:43.600472 (+ 36us) server_negotiation.cc:407] Connection header received
0927 00:48:43.601182 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:43.601200 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:43.601582 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:43.601980 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:43.602779 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:43.603780 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:43.604788 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:43.605265 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:43.605435 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:43.607900 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:43.607906 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:43.607920 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:43.607947 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:43.609853 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:43.610303 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:43.610308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:43.610317 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:43.610421 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:43.610731 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:43.610735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:43.610738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:43.611319 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:43.611506 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:43.611793 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 00:48:43.612083 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":176,"threads_started":1}
I20260927 00:48:43.613982 11682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 946be0b9236e4f04909359384ffe4152 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "946be0b9236e4f04909359384ffe4152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946be0b9236e4f04909359384ffe4152" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42309 } } }
I20260927 00:48:43.613960 11683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 946be0b9236e4f04909359384ffe4152 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 946be0b9236e4f04909359384ffe4152. Latest consensus state: current_term: 1 leader_uuid: "946be0b9236e4f04909359384ffe4152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946be0b9236e4f04909359384ffe4152" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42309 } } }
I20260927 00:48:43.614109 11683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 946be0b9236e4f04909359384ffe4152 [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:43.614109 11682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 946be0b9236e4f04909359384ffe4152 [sys.catalog]: This master's current role is: LEADER
I20260927 00:48:43.614521 11695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:48:43.614609 11676 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 00:48:43.621354 11695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:48:43.628168 11695 catalog_manager.cc:1347] Generated new cluster ID: 9c6df4c6af7045638f00f96a66287832
I20260927 00:48:43.628268 11695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:48:43.657254 11695 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:48:43.658505 11695 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:48:43.664140 11695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 946be0b9236e4f04909359384ffe4152: Generated new TSK 0
I20260927 00:48:43.664899 11695 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:48:44.432122 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38399/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42309
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:44.587906 11721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:44.588342 11721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:44.588435 11721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:44.594838 11721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:44.595000 11721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:44.595125 11721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:48:44.601279 11721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:38399/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:48:44.602591 11721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:44.603785 11721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:44.614423 11729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:44.614444 11731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:44.614775 11728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:44.615330 11721 server_base.cc:1034] running on GCE node
I20260927 00:48:44.615792 11721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:44.616645 11721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:44.617848 11721 hybrid_clock.cc:648] HybridClock initialized: now 1790470124617811 us; error 48 us; skew 500 ppm
Sep 27 00:48:44 dist-test-slave-86rs krb5kdc[11164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:44.621537 11721 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:48:44.622704 11721 webserver.cc:460] Webserver started at http://127.0.98.1:44993/ using document root <none> and password file <none>
I20260927 00:48:44.623493 11721 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:44.623644 11721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:44.623943 11721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:48:44 dist-test-slave-86rs krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:48:44.651131 11721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-0/data/instance:
uuid: "4e37985d61604f14b0a660afa9ed6dfa"
format_stamp: "Formatted at 2026-09-27 00:48:44 on dist-test-slave-86rs"
server_key: "579ca6c8a8aba8fbea8734414d6d7e57"
server_key_iv: "b3a5fa15e24231e92f902397e71e65c7"
server_key_version: "kuduclusterkey@0"
I20260927 00:48:44.652094 11721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4e37985d61604f14b0a660afa9ed6dfa"
format_stamp: "Formatted at 2026-09-27 00:48:44 on dist-test-slave-86rs"
server_key: "579ca6c8a8aba8fbea8734414d6d7e57"
server_key_iv: "b3a5fa15e24231e92f902397e71e65c7"
server_key_version: "kuduclusterkey@0"
I20260927 00:48:44.659067 11721 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260927 00:48:44.683949 11740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:44.686058 11721 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260927 00:48:44.686326 11721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "4e37985d61604f14b0a660afa9ed6dfa"
format_stamp: "Formatted at 2026-09-27 00:48:44 on dist-test-slave-86rs"
server_key: "579ca6c8a8aba8fbea8734414d6d7e57"
server_key_iv: "b3a5fa15e24231e92f902397e71e65c7"
server_key_version: "kuduclusterkey@0"
I20260927 00:48:44.686594 11721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:44.711966 11721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:44.716256 11721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:44.716487 11721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:44.717386 11721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:48:44.718644 11721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:48:44.718719 11721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:44.718779 11721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:48:44.718868 11721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 00:48:44.732259 11721 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39961
I20260927 00:48:44.733711 11721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:48:44.734740 11853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39961 every 8 connection(s)
I20260927 00:48:44.742525 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 11721
I20260927 00:48:44.742660 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:48:44.742996 392 external_mini_cluster.cc:1428] Setting key 7db68ce2828182d1c0ad1e6b6747547d
Sep 27 00:48:44 dist-test-slave-86rs krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 00:48:44 dist-test-slave-86rs krb5kdc[11164](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790470124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:48:44.756772 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:44.736498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35795 (local address 127.0.98.62:42309)
0927 00:48:44.739812 (+ 3314us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:44.739817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:44.741370 (+ 1553us) server_negotiation.cc:407] Connection header received
0927 00:48:44.741619 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:44.741624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:44.741700 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:44.741800 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:44.743630 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:44.744816 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:44.746617 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:44.747031 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:44.747146 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:44.750671 (+ 3525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:44.750676 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:44.750680 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:44.750716 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:44.753533 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:44.754372 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:44.754376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:44.754378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:44.754494 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:44.755016 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:44.755020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:44.755022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:44.755329 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:44.755420 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:44.756353 (+ 933us) server_negotiation.cc:299] Negotiation successful
0927 00:48:44.756516 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3198,"thread_start_us":116,"threads_started":1}
I20260927 00:48:44.758742 11857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:44.736608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42309 (local address 127.0.98.1:35795)
0927 00:48:44.740134 (+ 3526us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:44.740208 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:44.741347 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:44.742034 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:44.742047 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:44.742497 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:44.743230 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:44.743249 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:44.744966 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:44.744969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:44.746462 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:44.746475 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:44.747240 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:44.747245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:44.747485 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:44.748284 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:44.748312 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:44.750379 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:44.753719 (+ 3340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:44.753729 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:44.753752 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:44.754217 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:44.754641 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:44.754645 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:44.754647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:44.754878 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:44.755517 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:44.755524 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:44.755947 (+ 423us) client_negotiation.cc:769] Sending connection context
0927 00:48:44.757065 (+ 1118us) client_negotiation.cc:240] Negotiation successful
0927 00:48:44.757497 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3257,"thread_start_us":123,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:48:44.760488 11854 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42309
I20260927 00:48:44.760792 11854 heartbeater.cc:464] Registering TS with master...
I20260927 00:48:44.761509 11854 heartbeater.cc:511] Master 127.0.98.62:42309 requested a full tablet report, sending...
I20260927 00:48:44.763592 11639 ts_manager.cc:194] Registered new tserver with Master: 4e37985d61604f14b0a660afa9ed6dfa (127.0.98.1:39961)
I20260927 00:48:44.765470 11639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:48:44.777627 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38399/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42309
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:48:44.935842 11860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:44.936256 11860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:44.936331 11860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:44.941974 11860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:44.942056 11860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:44.942121 11860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:48:44.947793 11860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:38399/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:48:44.949033 11860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:44.950129 11860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:44.961215 11866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:44.961215 11867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:44.961514 11869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:44.961800 11860 server_base.cc:1034] running on GCE node
I20260927 00:48:44.962227 11860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:44.962962 11860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:44.964155 11860 hybrid_clock.cc:648] HybridClock initialized: now 1790470124964115 us; error 53 us; skew 500 ppm
Sep 27 00:48:44 dist-test-slave-86rs krb5kdc[11164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:44.967557 11860 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:48:44.968969 11860 webserver.cc:460] Webserver started at http://127.0.98.2:42065/ using document root <none> and password file <none>
I20260927 00:48:44.969825 11860 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:44.969937 11860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:44.970242 11860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:48:44 dist-test-slave-86rs krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:48:44.988016 11860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-1/data/instance:
uuid: "e0bb9a11eff24e6d9f22d550e753e1a0"
format_stamp: "Formatted at 2026-09-27 00:48:44 on dist-test-slave-86rs"
server_key: "634cb4bdb3a67fd7b664c41ce25c8f73"
server_key_iv: "d5f83ef267d68a6b24ac57933f5d83bd"
server_key_version: "kuduclusterkey@0"
I20260927 00:48:45.000015 11860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e0bb9a11eff24e6d9f22d550e753e1a0"
format_stamp: "Formatted at 2026-09-27 00:48:44 on dist-test-slave-86rs"
server_key: "634cb4bdb3a67fd7b664c41ce25c8f73"
server_key_iv: "d5f83ef267d68a6b24ac57933f5d83bd"
server_key_version: "kuduclusterkey@0"
I20260927 00:48:45.006600 11860 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 00:48:45.033521 11876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:45.035588 11860 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.000s
I20260927 00:48:45.035773 11860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "e0bb9a11eff24e6d9f22d550e753e1a0"
format_stamp: "Formatted at 2026-09-27 00:48:44 on dist-test-slave-86rs"
server_key: "634cb4bdb3a67fd7b664c41ce25c8f73"
server_key_iv: "d5f83ef267d68a6b24ac57933f5d83bd"
server_key_version: "kuduclusterkey@0"
I20260927 00:48:45.035943 11860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:45.059986 11860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:45.067677 11860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:45.068576 11860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:45.069525 11860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:48:45.070748 11860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:48:45.072299 11860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:45.072463 11860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:48:45.072531 11860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:45.087580 11860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36877
I20260927 00:48:45.088984 11860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:48:45.091424 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 11860
I20260927 00:48:45.091560 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:48:45.091899 392 external_mini_cluster.cc:1428] Setting key 49669e97998c55fd9c4eee36c876a559
I20260927 00:48:45.093796 11989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36877 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 27 00:48:45 dist-test-slave-86rs krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:48:45.125186 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:45.096201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41491 (local address 127.0.98.62:42309)
0927 00:48:45.096400 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:45.096406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:45.099245 (+ 2839us) server_negotiation.cc:407] Connection header received
0927 00:48:45.102223 (+ 2978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:45.102228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:45.102304 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:45.102370 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:45.106467 (+ 4097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.107483 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.112370 (+ 4887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.112843 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.112955 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:45.119725 (+ 6770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:45.119731 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:45.119734 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:45.119774 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:45.121922 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:45.122790 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:45.122795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:45.122797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:45.122904 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:45.123446 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:45.123450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:45.123452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:45.123733 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:45.123838 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:45.124767 (+ 929us) server_negotiation.cc:299] Negotiation successful
0927 00:48:45.124948 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260927 00:48:45.127524 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:45.094593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42309 (local address 127.0.98.2:41491)
0927 00:48:45.099018 (+ 4425us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:45.099113 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:45.100460 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:45.102901 (+ 2441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:45.102912 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:45.103369 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:45.104173 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:45.104200 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.107701 (+ 3501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.107704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:45.109487 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:45.109504 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.113267 (+ 3763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.113272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:45.113564 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:45.117277 (+ 3713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:45.117308 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:45.119414 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:45.122142 (+ 2728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:45.122150 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:45.122176 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:45.122633 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:45.123043 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:45.123046 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:45.123048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:45.123304 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:45.123919 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:45.123927 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:45.124410 (+ 483us) client_negotiation.cc:769] Sending connection context
0927 00:48:45.125761 (+ 1351us) client_negotiation.cc:240] Negotiation successful
0927 00:48:45.126219 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4136,"thread_start_us":160,"threads_started":1}
I20260927 00:48:45.130149 11990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42309
I20260927 00:48:45.130455 11990 heartbeater.cc:464] Registering TS with master...
I20260927 00:48:45.131183 11990 heartbeater.cc:511] Master 127.0.98.62:42309 requested a full tablet report, sending...
I20260927 00:48:45.132691 11639 ts_manager.cc:194] Registered new tserver with Master: e0bb9a11eff24e6d9f22d550e753e1a0 (127.0.98.2:36877)
I20260927 00:48:45.133589 11639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41491
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:48:45.164729 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38399/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42309
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:48:45 dist-test-slave-86rs krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 00:48:45.333057 11995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:48:45.333555 11995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:48:45.333675 11995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:48:45.340190 11995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:48:45.340318 11995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:48:45.340435 11995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:48:45.349174 11995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:38399/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:48:45.350641 11995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:48:45.351758 11995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:48:45.361922 12001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:48:45.361928 12003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:45.362373 11995 server_base.cc:1034] running on GCE node
W20260927 00:48:45.366077 12000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:48:45.366628 11995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:48:45.367420 11995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:48:45.368600 11995 hybrid_clock.cc:648] HybridClock initialized: now 1790470125368565 us; error 52 us; skew 500 ppm
Sep 27 00:48:45 dist-test-slave-86rs krb5kdc[11164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:48:45.371662 11995 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:48:45.372779 11995 webserver.cc:460] Webserver started at http://127.0.98.3:40547/ using document root <none> and password file <none>
I20260927 00:48:45.373385 11995 fs_manager.cc:359] Metadata directory not provided
I20260927 00:48:45.373481 11995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:48:45.373728 11995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:48:45 dist-test-slave-86rs krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:48:45.396900 11995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-2/data/instance:
uuid: "586cb642d9c945f8b9f5b336bcf77522"
format_stamp: "Formatted at 2026-09-27 00:48:45 on dist-test-slave-86rs"
server_key: "6c75ea999a49ff439e758bf00fd4706b"
server_key_iv: "f968a531cf26a40fe52bba97efc9b942"
server_key_version: "kuduclusterkey@0"
I20260927 00:48:45.397809 11995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-2/wal/instance:
uuid: "586cb642d9c945f8b9f5b336bcf77522"
format_stamp: "Formatted at 2026-09-27 00:48:45 on dist-test-slave-86rs"
server_key: "6c75ea999a49ff439e758bf00fd4706b"
server_key_iv: "f968a531cf26a40fe52bba97efc9b942"
server_key_version: "kuduclusterkey@0"
I20260927 00:48:45.403285 11995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 00:48:45.428799 12010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:45.430686 11995 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260927 00:48:45.430860 11995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "586cb642d9c945f8b9f5b336bcf77522"
format_stamp: "Formatted at 2026-09-27 00:48:45 on dist-test-slave-86rs"
server_key: "6c75ea999a49ff439e758bf00fd4706b"
server_key_iv: "f968a531cf26a40fe52bba97efc9b942"
server_key_version: "kuduclusterkey@0"
I20260927 00:48:45.431010 11995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:48:45.457393 11995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:48:45.461234 11995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:48:45.461443 11995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:48:45.462141 11995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:48:45.463208 11995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:48:45.463268 11995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:48:45.463320 11995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:48:45.463392 11995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 00:48:45.474731 11995 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40521
I20260927 00:48:45.474885 12123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40521 every 8 connection(s)
I20260927 00:48:45.476079 11995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:48:45.485497 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 11995
I20260927 00:48:45.485641 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790470006085796-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:48:45.485997 392 external_mini_cluster.cc:1428] Setting key 465fc0b3b063d569b45fa1da25fe5a41
Sep 27 00:48:45 dist-test-slave-86rs krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:48:45.494540 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:45.478276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56275 (local address 127.0.98.62:42309)
0927 00:48:45.478421 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:45.478426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:45.479540 (+ 1114us) server_negotiation.cc:407] Connection header received
0927 00:48:45.480850 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:45.480854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:45.480915 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:45.480977 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:45.483331 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.484200 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.486262 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.486622 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.486700 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:45.489944 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:45.489949 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:45.489952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:45.489983 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:45.491731 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:45.492481 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:45.492486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:45.492488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:45.492607 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:45.493032 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:45.493036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:45.493037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:45.493278 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:45.493375 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:45.494197 (+ 822us) server_negotiation.cc:299] Negotiation successful
0927 00:48:45.494347 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:48:45.495558 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:45.478787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42309 (local address 127.0.98.3:56275)
0927 00:48:45.479365 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:45.479424 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:45.480576 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:45.481459 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:45.481469 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:45.481882 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:48:45.483106 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:45.483129 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.484416 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.484419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:45.486043 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:45.486111 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.486924 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.486931 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:45.487153 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:45.487967 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:48:45.487989 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:48:45.489750 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:48:45.491903 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:45.491910 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:45.491930 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:45.492343 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:45.492709 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:48:45.492712 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:48:45.492714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:48:45.492918 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:48:45.493390 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:48:45.493397 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:48:45.493887 (+ 490us) client_negotiation.cc:769] Sending connection context
0927 00:48:45.494096 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 00:48:45.494463 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":152,"threads_started":1}
I20260927 00:48:45.496985 12124 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42309
I20260927 00:48:45.497262 12124 heartbeater.cc:464] Registering TS with master...
I20260927 00:48:45.497867 12124 heartbeater.cc:511] Master 127.0.98.62:42309 requested a full tablet report, sending...
I20260927 00:48:45.499137 11639 ts_manager.cc:194] Registered new tserver with Master: 586cb642d9c945f8b9f5b336bcf77522 (127.0.98.3:40521)
I20260927 00:48:45.499953 11639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56275
I20260927 00:48:45.500090 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:48:45.515514 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:45.503192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43586 (local address 127.0.98.62:42309)
0927 00:48:45.503341 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:45.503346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:45.505560 (+ 2214us) server_negotiation.cc:407] Connection header received
0927 00:48:45.505711 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:45.505714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:45.505781 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:45.505888 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:48:45.506838 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.507650 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.508884 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.509214 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.509335 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:45.511164 (+ 1829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:48:45.511169 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:48:45.511172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:48:45.511206 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:48:45.513141 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:45.513914 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:45.513918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:45.513919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:45.514018 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:48:45.514368 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:48:45.514372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:48:45.514374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:48:45.514627 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:48:45.514763 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:45.515057 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 00:48:45.515197 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:48:45.519407 11639 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:48:45.524065 11639 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:48:45.548794 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:45.536032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36877 (local address 127.0.0.1:37640)
0927 00:48:45.539469 (+ 3437us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:45.539481 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:45.539548 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:45.540994 (+ 1446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:45.540997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:45.541008 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:45.541096 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:45.541101 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.543536 (+ 2435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.543540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:45.544940 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:45.544951 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.548158 (+ 3207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.548161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:45.548315 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:45.548364 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 00:48:45.548461 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 00:48:45.548608 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3379,"thread_start_us":85,"threads_started":1}
I20260927 00:48:45.548980 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:45.537431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38370 (local address 127.0.98.3:40521)
0927 00:48:45.541354 (+ 3923us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:45.541364 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:45.541385 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:48:45.541452 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:45.541456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:45.541685 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:45.542056 (+ 371us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:45.542191 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.543275 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.547273 (+ 3998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.548026 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.548133 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:45.548345 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:45.548699 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 00:48:45.548814 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3821,"thread_start_us":92,"threads_started":1}
I20260927 00:48:45.549371 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:45.536615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40521 (local address 127.0.0.1:38370)
0927 00:48:45.539831 (+ 3216us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:45.539843 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:45.540888 (+ 1045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:45.541827 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:45.541830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:45.541846 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:45.541928 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:45.541934 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.543407 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.543410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:45.547123 (+ 3713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:45.547128 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.548404 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.548408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:45.548491 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:45.548509 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 00:48:45.548567 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 00:48:45.548618 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3160,"mutex_wait_us":1546,"thread_start_us":90,"threads_started":1}
I20260927 00:48:45.550828 12140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:45.536684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37640 (local address 127.0.98.2:36877)
0927 00:48:45.540281 (+ 3597us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:45.540297 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:45.540321 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:48:45.540394 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:45.540399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:45.540597 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:45.540719 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:45.541207 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.543397 (+ 2190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.547087 (+ 3690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.548009 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.549497 (+ 1488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:45.550517 (+ 1020us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:45.550576 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 00:48:45.550683 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3458,"thread_start_us":107,"threads_started":1}
I20260927 00:48:45.552937 12058 tablet_service.cc:1467] Processing CreateTablet for tablet 91a522ac82b047b39f8fd0a3c315bda3 (DEFAULT_TABLE table=test-table [id=a7c6211b88c04b4f887da9c04ebbc3f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:48:45.553683 11924 tablet_service.cc:1467] Processing CreateTablet for tablet 91a522ac82b047b39f8fd0a3c315bda3 (DEFAULT_TABLE table=test-table [id=a7c6211b88c04b4f887da9c04ebbc3f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:48:45.554136 12058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a522ac82b047b39f8fd0a3c315bda3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:45.554713 11924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a522ac82b047b39f8fd0a3c315bda3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:45.566340 12144 tablet_bootstrap.cc:492] T 91a522ac82b047b39f8fd0a3c315bda3 P e0bb9a11eff24e6d9f22d550e753e1a0: Bootstrap starting.
Sep 27 00:48:45 dist-test-slave-86rs krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:48:45.570531 12144 tablet_bootstrap.cc:654] T 91a522ac82b047b39f8fd0a3c315bda3 P e0bb9a11eff24e6d9f22d550e753e1a0: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:45.571545 12144 log.cc:824] T 91a522ac82b047b39f8fd0a3c315bda3 P e0bb9a11eff24e6d9f22d550e753e1a0: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:45.574059 12144 tablet_bootstrap.cc:492] T 91a522ac82b047b39f8fd0a3c315bda3 P e0bb9a11eff24e6d9f22d550e753e1a0: No bootstrap required, opened a new log
I20260927 00:48:45.574282 12144 ts_tablet_manager.cc:1397] T 91a522ac82b047b39f8fd0a3c315bda3 P e0bb9a11eff24e6d9f22d550e753e1a0: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 00:48:45.578306 12144 raft_consensus.cc:348] T 91a522ac82b047b39f8fd0a3c315bda3 P e0bb9a11eff24e6d9f22d550e753e1a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e37985d61604f14b0a660afa9ed6dfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } } peers { permanent_uuid: "586cb642d9c945f8b9f5b336bcf77522" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40521 } } peers { permanent_uuid: "e0bb9a11eff24e6d9f22d550e753e1a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } }
I20260927 00:48:45.578608 12144 raft_consensus.cc:374] T 91a522ac82b047b39f8fd0a3c315bda3 P e0bb9a11eff24e6d9f22d550e753e1a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:45.578652 12144 raft_consensus.cc:729] T 91a522ac82b047b39f8fd0a3c315bda3 P e0bb9a11eff24e6d9f22d550e753e1a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0bb9a11eff24e6d9f22d550e753e1a0, State: Initialized, Role: FOLLOWER
I20260927 00:48:45.579468 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:45.535746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39961 (local address 127.0.0.1:47278)
0927 00:48:45.537984 (+ 2238us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:45.538037 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:45.538156 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:45.539103 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:45.539109 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:45.539131 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:45.539269 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:45.539274 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.545904 (+ 6630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.545907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:45.547016 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:45.547026 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.578968 (+ 31942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.578973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:45.579078 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:45.579104 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:48:45.579191 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 00:48:45.579301 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2050,"mutex_wait_us":5,"thread_start_us":107,"threads_started":1}
I20260927 00:48:45.579588 12144 consensus_queue.cc:260] T 91a522ac82b047b39f8fd0a3c315bda3 P e0bb9a11eff24e6d9f22d550e753e1a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e37985d61604f14b0a660afa9ed6dfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } } peers { permanent_uuid: "586cb642d9c945f8b9f5b336bcf77522" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40521 } } peers { permanent_uuid: "e0bb9a11eff24e6d9f22d550e753e1a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } }
I20260927 00:48:45.580757 12144 ts_tablet_manager.cc:1428] T 91a522ac82b047b39f8fd0a3c315bda3 P e0bb9a11eff24e6d9f22d550e753e1a0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:48:45.581187 11990 heartbeater.cc:503] Master 127.0.98.62:42309 was elected leader, sending a full tablet report...
I20260927 00:48:45.583252 12145 tablet_bootstrap.cc:492] T 91a522ac82b047b39f8fd0a3c315bda3 P 586cb642d9c945f8b9f5b336bcf77522: Bootstrap starting.
I20260927 00:48:45.586562 12145 tablet_bootstrap.cc:654] T 91a522ac82b047b39f8fd0a3c315bda3 P 586cb642d9c945f8b9f5b336bcf77522: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:45.587635 12145 log.cc:824] T 91a522ac82b047b39f8fd0a3c315bda3 P 586cb642d9c945f8b9f5b336bcf77522: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:45.588503 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:45.536891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47278 (local address 127.0.98.1:39961)
0927 00:48:45.538478 (+ 1587us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:45.538486 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:45.538509 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:48:45.538585 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:45.538590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:45.538799 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:45.538927 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:45.544599 (+ 5672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.545757 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.571859 (+ 26102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.578820 (+ 6961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.578965 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:45.579139 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:45.587496 (+ 8357us) server_negotiation.cc:299] Negotiation successful
0927 00:48:45.588295 (+ 799us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1481,"thread_start_us":79,"threads_started":1}
W20260927 00:48:45.590796 11991 tablet.cc:2369] T 91a522ac82b047b39f8fd0a3c315bda3 P e0bb9a11eff24e6d9f22d550e753e1a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:48:45.593236 11775 tablet_service.cc:1467] Processing CreateTablet for tablet 91a522ac82b047b39f8fd0a3c315bda3 (DEFAULT_TABLE table=test-table [id=a7c6211b88c04b4f887da9c04ebbc3f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:48:45.594676 11775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a522ac82b047b39f8fd0a3c315bda3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:48:45.604063 12149 tablet_bootstrap.cc:492] T 91a522ac82b047b39f8fd0a3c315bda3 P 4e37985d61604f14b0a660afa9ed6dfa: Bootstrap starting.
I20260927 00:48:45.606760 12149 tablet_bootstrap.cc:654] T 91a522ac82b047b39f8fd0a3c315bda3 P 4e37985d61604f14b0a660afa9ed6dfa: Neither blocks nor log segments found. Creating new log.
I20260927 00:48:45.607791 12149 log.cc:824] T 91a522ac82b047b39f8fd0a3c315bda3 P 4e37985d61604f14b0a660afa9ed6dfa: Log is configured to *not* fsync() on all Append() calls
I20260927 00:48:45.610311 12149 tablet_bootstrap.cc:492] T 91a522ac82b047b39f8fd0a3c315bda3 P 4e37985d61604f14b0a660afa9ed6dfa: No bootstrap required, opened a new log
I20260927 00:48:45.610545 12149 ts_tablet_manager.cc:1397] T 91a522ac82b047b39f8fd0a3c315bda3 P 4e37985d61604f14b0a660afa9ed6dfa: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 00:48:45.613968 12145 tablet_bootstrap.cc:492] T 91a522ac82b047b39f8fd0a3c315bda3 P 586cb642d9c945f8b9f5b336bcf77522: No bootstrap required, opened a new log
I20260927 00:48:45.614195 12145 ts_tablet_manager.cc:1397] T 91a522ac82b047b39f8fd0a3c315bda3 P 586cb642d9c945f8b9f5b336bcf77522: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.000s
I20260927 00:48:45.615942 12149 raft_consensus.cc:348] T 91a522ac82b047b39f8fd0a3c315bda3 P 4e37985d61604f14b0a660afa9ed6dfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e37985d61604f14b0a660afa9ed6dfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } } peers { permanent_uuid: "586cb642d9c945f8b9f5b336bcf77522" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40521 } } peers { permanent_uuid: "e0bb9a11eff24e6d9f22d550e753e1a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } }
I20260927 00:48:45.616901 12149 raft_consensus.cc:374] T 91a522ac82b047b39f8fd0a3c315bda3 P 4e37985d61604f14b0a660afa9ed6dfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:45.617010 12149 raft_consensus.cc:729] T 91a522ac82b047b39f8fd0a3c315bda3 P 4e37985d61604f14b0a660afa9ed6dfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e37985d61604f14b0a660afa9ed6dfa, State: Initialized, Role: FOLLOWER
I20260927 00:48:45.617647 12149 consensus_queue.cc:260] T 91a522ac82b047b39f8fd0a3c315bda3 P 4e37985d61604f14b0a660afa9ed6dfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e37985d61604f14b0a660afa9ed6dfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } } peers { permanent_uuid: "586cb642d9c945f8b9f5b336bcf77522" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40521 } } peers { permanent_uuid: "e0bb9a11eff24e6d9f22d550e753e1a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } }
I20260927 00:48:45.618562 12145 raft_consensus.cc:348] T 91a522ac82b047b39f8fd0a3c315bda3 P 586cb642d9c945f8b9f5b336bcf77522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e37985d61604f14b0a660afa9ed6dfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } } peers { permanent_uuid: "586cb642d9c945f8b9f5b336bcf77522" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40521 } } peers { permanent_uuid: "e0bb9a11eff24e6d9f22d550e753e1a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } }
I20260927 00:48:45.618834 11854 heartbeater.cc:503] Master 127.0.98.62:42309 was elected leader, sending a full tablet report...
I20260927 00:48:45.618919 12149 ts_tablet_manager.cc:1428] T 91a522ac82b047b39f8fd0a3c315bda3 P 4e37985d61604f14b0a660afa9ed6dfa: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260927 00:48:45.618970 12145 raft_consensus.cc:374] T 91a522ac82b047b39f8fd0a3c315bda3 P 586cb642d9c945f8b9f5b336bcf77522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:48:45.619087 12145 raft_consensus.cc:729] T 91a522ac82b047b39f8fd0a3c315bda3 P 586cb642d9c945f8b9f5b336bcf77522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 586cb642d9c945f8b9f5b336bcf77522, State: Initialized, Role: FOLLOWER
I20260927 00:48:45.619642 12145 consensus_queue.cc:260] T 91a522ac82b047b39f8fd0a3c315bda3 P 586cb642d9c945f8b9f5b336bcf77522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e37985d61604f14b0a660afa9ed6dfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } } peers { permanent_uuid: "586cb642d9c945f8b9f5b336bcf77522" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40521 } } peers { permanent_uuid: "e0bb9a11eff24e6d9f22d550e753e1a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } }
I20260927 00:48:45.620985 12145 ts_tablet_manager.cc:1428] T 91a522ac82b047b39f8fd0a3c315bda3 P 586cb642d9c945f8b9f5b336bcf77522: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260927 00:48:45.621423 12124 heartbeater.cc:503] Master 127.0.98.62:42309 was elected leader, sending a full tablet report...
I20260927 00:48:45.684353 12151 raft_consensus.cc:482] T 91a522ac82b047b39f8fd0a3c315bda3 P 4e37985d61604f14b0a660afa9ed6dfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:48:45.684619 12151 raft_consensus.cc:504] T 91a522ac82b047b39f8fd0a3c315bda3 P 4e37985d61604f14b0a660afa9ed6dfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e37985d61604f14b0a660afa9ed6dfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } } peers { permanent_uuid: "586cb642d9c945f8b9f5b336bcf77522" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40521 } } peers { permanent_uuid: "e0bb9a11eff24e6d9f22d550e753e1a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } }
I20260927 00:48:45.686050 12151 leader_election.cc:288] T 91a522ac82b047b39f8fd0a3c315bda3 P 4e37985d61604f14b0a660afa9ed6dfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 586cb642d9c945f8b9f5b336bcf77522 (127.0.98.3:40521), e0bb9a11eff24e6d9f22d550e753e1a0 (127.0.98.2:36877)
I20260927 00:48:45.695816 12140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:45.686898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54099 (local address 127.0.98.2:36877)
0927 00:48:45.687033 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:45.687038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:45.689578 (+ 2540us) server_negotiation.cc:407] Connection header received
0927 00:48:45.691260 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:45.691264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:45.691315 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:45.691360 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:45.691783 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.692636 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.694449 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.695040 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.695128 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:45.695205 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:45.695559 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 00:48:45.695670 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:48:45.696594 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:45.686472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36877 (local address 127.0.98.1:54099)
0927 00:48:45.689464 (+ 2992us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:45.689481 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:45.691136 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:45.691491 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:45.691494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:45.691508 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:45.691655 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:45.691662 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.692835 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.692838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:45.694290 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:45.694304 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.695310 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.695313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:45.695413 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:45.695435 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:48:45.696304 (+ 869us) client_negotiation.cc:240] Negotiation successful
0927 00:48:45.696396 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2908,"thread_start_us":121,"threads_started":1}
I20260927 00:48:45.697499 11944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a522ac82b047b39f8fd0a3c315bda3" candidate_uuid: "4e37985d61604f14b0a660afa9ed6dfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0bb9a11eff24e6d9f22d550e753e1a0" is_pre_election: true
I20260927 00:48:45.697911 11944 raft_consensus.cc:2454] T 91a522ac82b047b39f8fd0a3c315bda3 P e0bb9a11eff24e6d9f22d550e753e1a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e37985d61604f14b0a660afa9ed6dfa in term 0.
I20260927 00:48:45.699208 11742 leader_election.cc:302] T 91a522ac82b047b39f8fd0a3c315bda3 P 4e37985d61604f14b0a660afa9ed6dfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e37985d61604f14b0a660afa9ed6dfa, e0bb9a11eff24e6d9f22d550e753e1a0; no voters:
I20260927 00:48:45.699585 12151 raft_consensus.cc:2781] T 91a522ac82b047b39f8fd0a3c315bda3 P 4e37985d61604f14b0a660afa9ed6dfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:48:45.699672 12151 raft_consensus.cc:482] T 91a522ac82b047b39f8fd0a3c315bda3 P 4e37985d61604f14b0a660afa9ed6dfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:48:45.699714 12151 raft_consensus.cc:3037] T 91a522ac82b047b39f8fd0a3c315bda3 P 4e37985d61604f14b0a660afa9ed6dfa [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:45.701273 12151 raft_consensus.cc:504] T 91a522ac82b047b39f8fd0a3c315bda3 P 4e37985d61604f14b0a660afa9ed6dfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e37985d61604f14b0a660afa9ed6dfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } } peers { permanent_uuid: "586cb642d9c945f8b9f5b336bcf77522" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40521 } } peers { permanent_uuid: "e0bb9a11eff24e6d9f22d550e753e1a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } }
I20260927 00:48:45.701524 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:45.687611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45533 (local address 127.0.98.3:40521)
0927 00:48:45.687759 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:45.687764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:45.690997 (+ 3233us) server_negotiation.cc:407] Connection header received
0927 00:48:45.691128 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:45.691132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:45.691190 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:45.691297 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:48:45.697845 (+ 6548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.698648 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.700163 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.700801 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.700890 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:45.700968 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:45.701310 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 00:48:45.701393 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:48:45.701709 12151 leader_election.cc:288] T 91a522ac82b047b39f8fd0a3c315bda3 P 4e37985d61604f14b0a660afa9ed6dfa [CANDIDATE]: Term 1 election: Requested vote from peers 586cb642d9c945f8b9f5b336bcf77522 (127.0.98.3:40521), e0bb9a11eff24e6d9f22d550e753e1a0 (127.0.98.2:36877)
I20260927 00:48:45.701951 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:45.687251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40521 (local address 127.0.98.1:45533)
0927 00:48:45.690898 (+ 3647us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:45.690913 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:45.691029 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:45.697410 (+ 6381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:48:45.697413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:48:45.697446 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:48:45.697580 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:45.697586 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.698792 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.698795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:45.699992 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:48:45.700005 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.701062 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.701066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:48:45.701161 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:45.701182 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:48:45.701721 (+ 539us) client_negotiation.cc:240] Negotiation successful
0927 00:48:45.701803 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3564,"thread_start_us":80,"threads_started":1}
I20260927 00:48:45.702667 11944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a522ac82b047b39f8fd0a3c315bda3" candidate_uuid: "4e37985d61604f14b0a660afa9ed6dfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0bb9a11eff24e6d9f22d550e753e1a0"
I20260927 00:48:45.702744 12078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a522ac82b047b39f8fd0a3c315bda3" candidate_uuid: "4e37985d61604f14b0a660afa9ed6dfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "586cb642d9c945f8b9f5b336bcf77522" is_pre_election: true
I20260927 00:48:45.702817 11944 raft_consensus.cc:3037] T 91a522ac82b047b39f8fd0a3c315bda3 P e0bb9a11eff24e6d9f22d550e753e1a0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:45.703054 12078 raft_consensus.cc:2454] T 91a522ac82b047b39f8fd0a3c315bda3 P 586cb642d9c945f8b9f5b336bcf77522 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e37985d61604f14b0a660afa9ed6dfa in term 0.
I20260927 00:48:45.703370 12077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a522ac82b047b39f8fd0a3c315bda3" candidate_uuid: "4e37985d61604f14b0a660afa9ed6dfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "586cb642d9c945f8b9f5b336bcf77522"
I20260927 00:48:45.703464 12077 raft_consensus.cc:3037] T 91a522ac82b047b39f8fd0a3c315bda3 P 586cb642d9c945f8b9f5b336bcf77522 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:48:45.704406 11944 raft_consensus.cc:2454] T 91a522ac82b047b39f8fd0a3c315bda3 P e0bb9a11eff24e6d9f22d550e753e1a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e37985d61604f14b0a660afa9ed6dfa in term 1.
I20260927 00:48:45.704706 12077 raft_consensus.cc:2454] T 91a522ac82b047b39f8fd0a3c315bda3 P 586cb642d9c945f8b9f5b336bcf77522 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e37985d61604f14b0a660afa9ed6dfa in term 1.
I20260927 00:48:45.704784 11742 leader_election.cc:302] T 91a522ac82b047b39f8fd0a3c315bda3 P 4e37985d61604f14b0a660afa9ed6dfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e37985d61604f14b0a660afa9ed6dfa, e0bb9a11eff24e6d9f22d550e753e1a0; no voters:
I20260927 00:48:45.705011 12151 raft_consensus.cc:2781] T 91a522ac82b047b39f8fd0a3c315bda3 P 4e37985d61604f14b0a660afa9ed6dfa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:48:45.705227 12151 raft_consensus.cc:686] T 91a522ac82b047b39f8fd0a3c315bda3 P 4e37985d61604f14b0a660afa9ed6dfa [term 1 LEADER]: Becoming Leader. State: Replica: 4e37985d61604f14b0a660afa9ed6dfa, State: Running, Role: LEADER
I20260927 00:48:45.705603 12151 consensus_queue.cc:237] T 91a522ac82b047b39f8fd0a3c315bda3 P 4e37985d61604f14b0a660afa9ed6dfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e37985d61604f14b0a660afa9ed6dfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } } peers { permanent_uuid: "586cb642d9c945f8b9f5b336bcf77522" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40521 } } peers { permanent_uuid: "e0bb9a11eff24e6d9f22d550e753e1a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } }
I20260927 00:48:45.709307 11638 catalog_manager.cc:5495] T 91a522ac82b047b39f8fd0a3c315bda3 P 4e37985d61604f14b0a660afa9ed6dfa reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e37985d61604f14b0a660afa9ed6dfa (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4e37985d61604f14b0a660afa9ed6dfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e37985d61604f14b0a660afa9ed6dfa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "586cb642d9c945f8b9f5b336bcf77522" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0bb9a11eff24e6d9f22d550e753e1a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 } health_report { overall_health: UNKNOWN } } }
W20260927 00:48:45.728454 12125 tablet.cc:2369] T 91a522ac82b047b39f8fd0a3c315bda3 P 586cb642d9c945f8b9f5b336bcf77522: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:48:45.741015 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:45.735552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47294 (local address 127.0.98.1:39961)
0927 00:48:45.735677 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:48:45.735682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:48:45.735753 (+ 71us) server_negotiation.cc:407] Connection header received
0927 00:48:45.735964 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:48:45.735968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:48:45.736059 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:48:45.736166 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:48:45.736664 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.737507 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.738785 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:48:45.739113 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:48:45.739214 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:48:45.739655 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:48:45.740194 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:48:45.740383 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 00:48:45.740816 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 00:48:45.740878 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 00:48:45.741456 11855 tablet.cc:2369] T 91a522ac82b047b39f8fd0a3c315bda3 P 4e37985d61604f14b0a660afa9ed6dfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:48:45.752497 12077 raft_consensus.cc:1261] T 91a522ac82b047b39f8fd0a3c315bda3 P 586cb642d9c945f8b9f5b336bcf77522 [term 1 FOLLOWER]: Refusing update from remote peer 4e37985d61604f14b0a660afa9ed6dfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:48:45.752497 11944 raft_consensus.cc:1261] T 91a522ac82b047b39f8fd0a3c315bda3 P e0bb9a11eff24e6d9f22d550e753e1a0 [term 1 FOLLOWER]: Refusing update from remote peer 4e37985d61604f14b0a660afa9ed6dfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:48:45.753432 12156 consensus_queue.cc:1035] T 91a522ac82b047b39f8fd0a3c315bda3 P 4e37985d61604f14b0a660afa9ed6dfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "586cb642d9c945f8b9f5b336bcf77522" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:48:45.753708 12151 consensus_queue.cc:1035] T 91a522ac82b047b39f8fd0a3c315bda3 P 4e37985d61604f14b0a660afa9ed6dfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0bb9a11eff24e6d9f22d550e753e1a0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:48:45.771656 12160 mvcc.cc:205] Tried to move back new op lower bound from 7333765635073454080 to 7333765634894102528. Current Snapshot: MvccSnapshot[applied={T|T < 7333765635073454080}]
I20260927 00:48:45.774194 12159 mvcc.cc:205] Tried to move back new op lower bound from 7333765635073454080 to 7333765634894102528. Current Snapshot: MvccSnapshot[applied={T|T < 7333765635073454080}]
I20260927 00:48:45.785866 12162 mvcc.cc:205] Tried to move back new op lower bound from 7333765635073454080 to 7333765634894102528. Current Snapshot: MvccSnapshot[applied={T|T < 7333765635073454080}]
I20260927 00:48:45.796612 11639 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:48:45.796840 11639 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:48:45.799736 11639 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 946be0b9236e4f04909359384ffe4152: Sending DeleteTablet for 3 replicas of tablet 91a522ac82b047b39f8fd0a3c315bda3
I20260927 00:48:45.800616 11775 tablet_service.cc:1514] Processing DeleteTablet for tablet 91a522ac82b047b39f8fd0a3c315bda3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:48:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47278
I20260927 00:48:45.801343 12058 tablet_service.cc:1514] Processing DeleteTablet for tablet 91a522ac82b047b39f8fd0a3c315bda3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:48:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38370
I20260927 00:48:45.801759 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 11721
I20260927 00:48:45.801867 11924 tablet_service.cc:1514] Processing DeleteTablet for tablet 91a522ac82b047b39f8fd0a3c315bda3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:48:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37640
I20260927 00:48:45.810029 12170 tablet_replica.cc:330] T 91a522ac82b047b39f8fd0a3c315bda3 P e0bb9a11eff24e6d9f22d550e753e1a0: stopping tablet replica
I20260927 00:48:45.810585 12170 raft_consensus.cc:2229] T 91a522ac82b047b39f8fd0a3c315bda3 P e0bb9a11eff24e6d9f22d550e753e1a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:48:45.811121 12170 raft_consensus.cc:2258] T 91a522ac82b047b39f8fd0a3c315bda3 P e0bb9a11eff24e6d9f22d550e753e1a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:48:45.811721 12169 tablet_replica.cc:330] T 91a522ac82b047b39f8fd0a3c315bda3 P 586cb642d9c945f8b9f5b336bcf77522: stopping tablet replica
I20260927 00:48:45.813527 12170 ts_tablet_manager.cc:1905] T 91a522ac82b047b39f8fd0a3c315bda3 P e0bb9a11eff24e6d9f22d550e753e1a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:48:45.816771 12170 ts_tablet_manager.cc:1918] T 91a522ac82b047b39f8fd0a3c315bda3 P e0bb9a11eff24e6d9f22d550e753e1a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260927 00:48:45.817363 11624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39961 (error 108)
I20260927 00:48:45.818598 12169 raft_consensus.cc:2229] T 91a522ac82b047b39f8fd0a3c315bda3 P 586cb642d9c945f8b9f5b336bcf77522 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:48:45.818926 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 11860
I20260927 00:48:45.822624 12169 raft_consensus.cc:2258] T 91a522ac82b047b39f8fd0a3c315bda3 P 586cb642d9c945f8b9f5b336bcf77522 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:48:45.829355 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:48:45.829017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39961 (local address 127.0.0.1:47310)
0927 00:48:45.829155 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:45.829226 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 00:48:45.829713 11624 catalog_manager.cc:4584] TS 4e37985d61604f14b0a660afa9ed6dfa (127.0.98.1:39961): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 91a522ac82b047b39f8fd0a3c315bda3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39961: connect: Connection refused (error 111)
I20260927 00:48:45.830137 12169 ts_tablet_manager.cc:1905] T 91a522ac82b047b39f8fd0a3c315bda3 P 586cb642d9c945f8b9f5b336bcf77522: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 00:48:45.830276 12139 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:48:45.829417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36877 (local address 127.0.0.1:37642)
0927 00:48:45.829507 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:45.829524 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:45.829620 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:45.830188 (+ 568us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36877: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 00:48:45.830966 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 11995
W20260927 00:48:45.838692 12139 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:48:45.838051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40521 (local address 127.0.0.1:38378)
0927 00:48:45.838262 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 00:48:45.838278 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:48:45.838379 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:48:45.838580 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:40521: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":123}
I20260927 00:48:45.839373 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 11608
I20260927 00:48:45.846122 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 00:48:46.191439 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 00:48:46.191670 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 00:48:46.537362 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 00:48:46.537 UTC [11194] LOG: received smart shutdown request
2026-09-27 00:48:46.541 UTC [11194] LOG: background worker "logical replication launcher" (PID 11203) exited with exit code 1
2026-09-27 00:48:46.541 UTC [11198] LOG: shutting down
2026-09-27 00:48:46.719 UTC [11194] LOG: database system is shut down
2026-09-27T00:48:46Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50160 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:48:46 dist-test-slave-86rs krb5kdc[12180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:48:46 dist-test-slave-86rs krb5kdc[12180](info): set up 2 sockets
Sep 27 00:48:46 dist-test-slave-86rs krb5kdc[12180](info): commencing operation
krb5kdc: starting...
W20260927 00:48:47.888350 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:48:47 dist-test-slave-86rs krb5kdc[12180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:48:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:48:47Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 00:48:48.063560 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskMaietF/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/postgres -l logfile start
2026-09-27 00:48:50.725 UTC [12210] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 00:48:50.725 UTC [12210] LOG: listening on IPv4 address "127.0.98.20", port 50171
2026-09-27 00:48:50.728 UTC [12210] LOG: listening on Unix socket "/tmp/.s.PGSQL.50171"
2026-09-27 00:48:50.752 UTC [12213] LOG: database system was shut down at 2026-09-27 00:48:49 UTC
2026-09-27 00:48:50.756 UTC [12210] LOG: database system is ready to accept connections
I20260927 00:48:51.712194 392 mini_postgres.cc:96] Postgres bound to 50171
2026-09-27 00:48:51.718 UTC [12221] FATAL: database "slave" does not exist
127.0.98.20:50171 - accepting connections
I20260927 00:48:51.719300 392 mini_ranger.cc:161] Starting Ranger...
I20260927 00:48:51.734761 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 00:48:52.295079 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 00:48:52.295183 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ranger-admin
2026-09-27 00:48:52,594 [I] DB FLAVOR :POSTGRES
2026-09-27 00:48:52,594 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 00:48:52,594 [I] Checking connection..
2026-09-27 00:48:52,594 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 00:48:52,867 [I] Checking connection passed.
2026-09-27 00:48:52,867 [I] --------- Verifying version history table ---------
2026-09-27 00:48:52,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:48:53,130 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 00:48:53,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:48:53,382 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 00:48:53,382 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 00:48:53,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50171/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 00:48:53.623 UTC [12285] WARNING: there is no transaction in progress
2026-09-27 00:48:53,630 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 00:48:53,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:48:53,899 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 00:48:53,900 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 00:48:53,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 00:48:54,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 00:48:54,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-86rs.c.gcp-upstream.internal','N') ;"
2026-09-27 00:48:54,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 00:48:54,944 [I] Table x_portal_user does not exist in database ranger
2026-09-27 00:48:54,944 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 00:48:55,213 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 00:48:55,213 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 00:48:55,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 00:48:55,727 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 00:48:55,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50171/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 00:48:56.758 UTC [12393] WARNING: there is no transaction in progress
2026-09-27 00:48:56.776 UTC [12393] WARNING: there is no transaction in progress
2026-09-27 00:48:56.795 UTC [12393] WARNING: there is no transaction in progress
2026-09-27 00:48:56.811 UTC [12393] WARNING: there is no transaction in progress
2026-09-27 00:48:56.827 UTC [12393] WARNING: there is no transaction in progress
2026-09-27 00:48:56.845 UTC [12393] WARNING: there is no transaction in progress
2026-09-27 00:48:56.905 UTC [12393] WARNING: there is no transaction in progress
2026-09-27 00:48:56.912 UTC [12393] WARNING: there is no transaction in progress
2026-09-27 00:48:56.927 UTC [12393] WARNING: there is no transaction in progress
2026-09-27 00:48:56.938 UTC [12393] WARNING: there is no transaction in progress
2026-09-27 00:48:56.949 UTC [12393] WARNING: there is no transaction in progress
2026-09-27 00:48:56.965 UTC [12393] WARNING: there is no transaction in progress
2026-09-27 00:48:56.975 UTC [12393] WARNING: there is no transaction in progress
2026-09-27 00:48:56.983 UTC [12393] WARNING: there is no transaction in progress
2026-09-27 00:48:56.990 UTC [12393] WARNING: there is no transaction in progress
2026-09-27 00:48:56.998 UTC [12393] WARNING: there is no transaction in progress
2026-09-27 00:48:57,718 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 00:48:57,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 00:48:57,971 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 00:48:57,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 00:48:58,244 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 00:48:58,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 00:48:58,501 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 00:48:58,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 00:48:58,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 00:48:59,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-86rs.c.gcp-upstream.internal';"
2026-09-27 00:48:59,251 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 00:48:59,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 00:48:59,511 [I] DB_PATCHES have already been applied
I20260927 00:48:59.517923 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskMaietF/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 00:48:59.518023 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 00:48:59.521610 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:33641
Sep 27, 2026 12:48:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:48:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:48:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 12:48:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 12:48:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 12:49:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 12:49:00 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 00:49:00.654616 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.935s user 0.000s sys 0.010s
Sep 27, 2026 12:49:21 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 12:49:21 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 12:49:21 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 12:49:22 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 12:49:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 12:49:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 12:49:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 12:49:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 12:49:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 12:49:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 12:49:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 12:49:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 12:49:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 12:49:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 12:49:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 12:49:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 12:49:22 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 12:49:22 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 00:49:23.610751 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 00:49:23 dist-test-slave-86rs krb5kdc[12180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470163, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 00:49:23.699034 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 00:49:23.708920 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 00:49:24.669796 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 00:49:24.669895 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ranger-kms
I20260927 00:49:24.669921 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:50171
2026-09-27 00:49:24,858 [I] DB FLAVOR :POSTGRES
2026-09-27 00:49:24,859 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 00:49:24,859 [I] Checking connection
2026-09-27 00:49:24,859 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50171/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 00:49:25,140 [I] connection success
2026-09-27 00:49:25,141 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 00:49:25,141 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 00:49:25,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50171/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 00:49:25,403 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 00:49:25,403 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 00:49:25,403 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 00:49:25,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50171/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 00:49:25,680 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 00:49:26.184521 392 mini_ranger_kms.cc:310] Created kms service
I20260927 00:49:26.383189 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 00:49:26.435585 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 00:49:26 dist-test-slave-86rs krb5kdc[12180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470163, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:49:26.888715 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 00:49:26.888882 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskMaietF/build/debug/bin/postgresql.jar:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskMaietF/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 00:49:26.888902 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 00:49:26.892856 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12574
I20260927 00:49:26.892956 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:51685
Sep 27, 2026 12:49:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:49:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:49:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 12:49:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 12:49:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 12:49:28 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 00:49:29.187218 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.488s user 0.006s sys 0.162s
I20260927 00:49:29.187320 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 00:49:29.187398 392 mini_ranger_kms.cc:401] 127.0.98.20:51685/kms/v1/keys
Sep 27 00:49:29 dist-test-slave-86rs krb5kdc[12180](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790470169, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 00:49:29 dist-test-slave-86rs krb5kdc[12180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470169, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 00:49:30 dist-test-slave-86rs krb5kdc[12180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470169, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 00:49:32 dist-test-slave-86rs krb5kdc[12180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:49:32.451120 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33621
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33621
--ranger_config_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51685/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:49:32.572307 12624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:49:32.572669 12624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:49:32.572773 12624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:49:32.577159 12624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:49:32.577266 12624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:49:32.577303 12624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:49:32.577346 12624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:49:32.577389 12624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:49:32.583493 12624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:51685/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33621
--ranger_config_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:49:32.584827 12624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:49:32.586170 12624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:49:32.596575 12630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:32.596575 12629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:32.596603 12632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:32.597940 12624 server_base.cc:1034] running on GCE node
I20260927 00:49:32.598681 12624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:49:32.600031 12624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:49:32.601258 12624 hybrid_clock.cc:648] HybridClock initialized: now 1790470172601231 us; error 51 us; skew 500 ppm
Sep 27 00:49:32 dist-test-slave-86rs krb5kdc[12180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:49:32.609849 12624 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:49:32.611034 12624 webserver.cc:460] Webserver started at http://127.0.98.62:39583/ using document root <none> and password file <none>
I20260927 00:49:32.611625 12624 fs_manager.cc:359] Metadata directory not provided
I20260927 00:49:32.611683 12624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:49:32.611914 12624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:49:32 dist-test-slave-86rs krb5kdc[12180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:49:32.666556 12624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "b2d0a9d6b7ea4b5a86384eb6585a1644"
format_stamp: "Formatted at 2026-09-27 00:49:32 on dist-test-slave-86rs"
server_key: "441697c4cd4553f177d363615394ed58"
server_key_iv: "40a8c7ef5fa5d8cd6353f159bf88b52e"
server_key_version: "kuduclusterkey@0"
I20260927 00:49:32.667272 12624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "b2d0a9d6b7ea4b5a86384eb6585a1644"
format_stamp: "Formatted at 2026-09-27 00:49:32 on dist-test-slave-86rs"
server_key: "441697c4cd4553f177d363615394ed58"
server_key_iv: "40a8c7ef5fa5d8cd6353f159bf88b52e"
server_key_version: "kuduclusterkey@0"
I20260927 00:49:32.671583 12624 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:49:32.686389 12639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:32.687930 12624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:49:32.688105 12624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "b2d0a9d6b7ea4b5a86384eb6585a1644"
format_stamp: "Formatted at 2026-09-27 00:49:32 on dist-test-slave-86rs"
server_key: "441697c4cd4553f177d363615394ed58"
server_key_iv: "40a8c7ef5fa5d8cd6353f159bf88b52e"
server_key_version: "kuduclusterkey@0"
I20260927 00:49:32.688310 12624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:49:32.718007 12624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:49:32.721290 12624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:49:32.721508 12624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:49:32.730104 12624 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33621
I20260927 00:49:32.730113 12691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33621 every 8 connection(s)
I20260927 00:49:32.731281 12624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:49:32.734346 12692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:49:32.740468 12692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644: Bootstrap starting.
I20260927 00:49:32.740586 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 12624
I20260927 00:49:32.740708 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:49:32.740983 392 external_mini_cluster.cc:1428] Setting key 6e3cbdeee76f79db5df9494b79bec772
I20260927 00:49:32.743484 12692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644: Neither blocks nor log segments found. Creating new log.
I20260927 00:49:32.744463 12692 log.cc:824] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644: Log is configured to *not* fsync() on all Append() calls
I20260927 00:49:32.746824 12692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644: No bootstrap required, opened a new log
Sep 27 00:49:32 dist-test-slave-86rs krb5kdc[12180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:49:32.751585 12692 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d0a9d6b7ea4b5a86384eb6585a1644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33621 } }
I20260927 00:49:32.751899 12692 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:49:32.752008 12692 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d0a9d6b7ea4b5a86384eb6585a1644, State: Initialized, Role: FOLLOWER
I20260927 00:49:32.752594 12692 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d0a9d6b7ea4b5a86384eb6585a1644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33621 } }
I20260927 00:49:32.752755 12692 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:49:32.752842 12692 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:49:32.752941 12692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:49:32.754142 12692 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d0a9d6b7ea4b5a86384eb6585a1644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33621 } }
I20260927 00:49:32.754542 12692 leader_election.cc:302] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2d0a9d6b7ea4b5a86384eb6585a1644; no voters:
I20260927 00:49:32.754899 12692 leader_election.cc:288] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:49:32.755052 12697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:49:32.755331 12697 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [term 1 LEADER]: Becoming Leader. State: Replica: b2d0a9d6b7ea4b5a86384eb6585a1644, State: Running, Role: LEADER
I20260927 00:49:32.755707 12697 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d0a9d6b7ea4b5a86384eb6585a1644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33621 } }
I20260927 00:49:32.756364 12692 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:49:32.756671 12695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:32.742546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60026 (local address 127.0.98.62:33621)
0927 00:49:32.743044 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:32.743063 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:32.743107 (+ 44us) server_negotiation.cc:407] Connection header received
0927 00:49:32.744053 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:32.744079 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:32.744562 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:32.745015 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:32.745854 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:32.746871 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:32.747814 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:32.748385 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:32.748569 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:32.751190 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:32.751196 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:32.751213 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:32.751245 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:32.753225 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:32.753782 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:32.753788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:32.753799 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:32.753907 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:32.754202 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:32.754207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:32.754209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:32.754698 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:32.754898 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:32.755327 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 00:49:32.755656 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":158,"threads_started":1}
I20260927 00:49:32.757076 12697 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2d0a9d6b7ea4b5a86384eb6585a1644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d0a9d6b7ea4b5a86384eb6585a1644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33621 } } }
I20260927 00:49:32.757222 12697 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [sys.catalog]: This master's current role is: LEADER
I20260927 00:49:32.757418 12697 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2d0a9d6b7ea4b5a86384eb6585a1644. Latest consensus state: current_term: 1 leader_uuid: "b2d0a9d6b7ea4b5a86384eb6585a1644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d0a9d6b7ea4b5a86384eb6585a1644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33621 } } }
I20260927 00:49:32.757506 12697 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [sys.catalog]: This master's current role is: LEADER
I20260927 00:49:32.757769 12705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:49:32.761178 12705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:49:32.761830 12692 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 00:49:32.780893 12705 catalog_manager.cc:1347] Generated new cluster ID: a85fbc09fdfd49329179f36eb4c2b391
I20260927 00:49:32.780995 12705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:49:32.801509 12705 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:49:32.803108 12705 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:49:32.811833 12705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644: Generated new TSK 0
I20260927 00:49:32.812935 12705 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:49:33.468036 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51685/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33621
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:49:33.617125 12737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:49:33.617538 12737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:49:33.617607 12737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:49:33.623908 12737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:49:33.624055 12737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:49:33.624183 12737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:49:33.632493 12737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:51685/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:49:33.634011 12737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:49:33.635437 12737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:49:33.648595 12744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:33.648830 12737 server_base.cc:1034] running on GCE node
W20260927 00:49:33.648633 12745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:33.648660 12747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:33.649494 12737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:49:33.650324 12737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:49:33.651526 12737 hybrid_clock.cc:648] HybridClock initialized: now 1790470173651488 us; error 58 us; skew 500 ppm
Sep 27 00:49:33 dist-test-slave-86rs krb5kdc[12180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:49:33.654723 12737 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:49:33.655890 12737 webserver.cc:460] Webserver started at http://127.0.98.1:35257/ using document root <none> and password file <none>
I20260927 00:49:33.656612 12737 fs_manager.cc:359] Metadata directory not provided
I20260927 00:49:33.656839 12737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:49:33.657115 12737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:49:33 dist-test-slave-86rs krb5kdc[12180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:49:33.690631 12737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/data/instance:
uuid: "e5cce0c6ed734684ae4fba815a1c2d19"
format_stamp: "Formatted at 2026-09-27 00:49:33 on dist-test-slave-86rs"
server_key: "4485da15a05be4b3fe72e0991c0a49ac"
server_key_iv: "635ef8849682614810c268de55e78315"
server_key_version: "kuduclusterkey@0"
I20260927 00:49:33.691499 12737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e5cce0c6ed734684ae4fba815a1c2d19"
format_stamp: "Formatted at 2026-09-27 00:49:33 on dist-test-slave-86rs"
server_key: "4485da15a05be4b3fe72e0991c0a49ac"
server_key_iv: "635ef8849682614810c268de55e78315"
server_key_version: "kuduclusterkey@0"
I20260927 00:49:33.697281 12737 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 00:49:33.740650 12756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:33.742452 12737 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.003s sys 0.000s
I20260927 00:49:33.742609 12737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "e5cce0c6ed734684ae4fba815a1c2d19"
format_stamp: "Formatted at 2026-09-27 00:49:33 on dist-test-slave-86rs"
server_key: "4485da15a05be4b3fe72e0991c0a49ac"
server_key_iv: "635ef8849682614810c268de55e78315"
server_key_version: "kuduclusterkey@0"
I20260927 00:49:33.742753 12737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:49:33.779819 12737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:49:33.783840 12737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:49:33.784114 12737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:49:33.784942 12737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:49:33.786134 12737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:49:33.786284 12737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:33.786429 12737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:49:33.786513 12737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:49:33 dist-test-slave-86rs krb5kdc[12180](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790470173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:49:33.806902 12737 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45583
I20260927 00:49:33.808334 12737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:49:33.809392 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 12737
I20260927 00:49:33.809521 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:49:33.809854 392 external_mini_cluster.cc:1428] Setting key 6eaff03f8a71ce99d458cab336206386
I20260927 00:49:33.809996 12869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45583 every 8 connection(s)
Sep 27 00:49:33 dist-test-slave-86rs krb5kdc[12180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:49:33.832398 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:33.812808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33621 (local address 127.0.98.1:39805)
0927 00:49:33.814621 (+ 1813us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:33.814677 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:33.815737 (+ 1060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:33.818427 (+ 2690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:33.818438 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:33.818837 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:49:33.819553 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:33.819573 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:33.820798 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:33.820802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:33.822246 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:33.822258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:33.823038 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:33.823042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:33.823260 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:33.824014 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:49:33.824036 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:49:33.825818 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:49:33.828523 (+ 2705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:33.828532 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:33.828557 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:33.828985 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:33.829363 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:33.829367 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:33.829371 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:33.829591 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:33.830145 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:49:33.830155 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:49:33.830571 (+ 416us) client_negotiation.cc:769] Sending connection context
0927 00:49:33.830819 (+ 248us) client_negotiation.cc:240] Negotiation successful
0927 00:49:33.831213 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1568,"thread_start_us":174,"threads_started":1}
I20260927 00:49:33.833170 12874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:33.813624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39805 (local address 127.0.98.62:33621)
0927 00:49:33.817816 (+ 4192us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:33.817825 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:33.817843 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:49:33.817895 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:33.817898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:33.817952 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:33.818048 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:33.819749 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:33.820638 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:33.822392 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:33.822741 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:33.822843 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:33.826071 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:33.826077 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:33.826081 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:33.826131 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:33.828355 (+ 2224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:33.829129 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:33.829135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:33.829139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:33.829247 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:33.829709 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:33.829712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:33.829715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:33.829980 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:33.830066 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:33.832848 (+ 2782us) server_negotiation.cc:299] Negotiation successful
0927 00:49:33.832995 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4097,"thread_start_us":84,"threads_started":1}
I20260927 00:49:33.835000 12870 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33621
I20260927 00:49:33.835271 12870 heartbeater.cc:464] Registering TS with master...
I20260927 00:49:33.835870 12870 heartbeater.cc:511] Master 127.0.98.62:33621 requested a full tablet report, sending...
I20260927 00:49:33.837852 12656 ts_manager.cc:194] Registered new tserver with Master: e5cce0c6ed734684ae4fba815a1c2d19 (127.0.98.1:45583)
I20260927 00:49:33.839465 12656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39805
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:49:33.875630 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51685/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33621
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:49:34.042515 12876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:49:34.043109 12876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:49:34.043259 12876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:49:34.051311 12876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:49:34.051512 12876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:49:34.051687 12876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:49:34.060833 12876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:51685/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:49:34.062757 12876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:49:34.064580 12876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:49:34.079504 12885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:34.079437 12883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:34.079428 12882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:34.081247 12876 server_base.cc:1034] running on GCE node
I20260927 00:49:34.081815 12876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:49:34.082679 12876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:49:34.086002 12876 hybrid_clock.cc:648] HybridClock initialized: now 1790470174085902 us; error 103 us; skew 500 ppm
Sep 27 00:49:34 dist-test-slave-86rs krb5kdc[12180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:49:34.090415 12876 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:49:34.092022 12876 webserver.cc:460] Webserver started at http://127.0.98.2:46681/ using document root <none> and password file <none>
I20260927 00:49:34.093017 12876 fs_manager.cc:359] Metadata directory not provided
I20260927 00:49:34.093205 12876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:49:34.093559 12876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:49:34 dist-test-slave-86rs krb5kdc[12180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:49:34.111923 12876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/data/instance:
uuid: "1e4417c04271412f83c037bda3a3883c"
format_stamp: "Formatted at 2026-09-27 00:49:34 on dist-test-slave-86rs"
server_key: "73e2eca86679923d0485099368e86fcd"
server_key_iv: "08ada63e7d043bf873febc2696f79abc"
server_key_version: "kuduclusterkey@0"
I20260927 00:49:34.112910 12876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1e4417c04271412f83c037bda3a3883c"
format_stamp: "Formatted at 2026-09-27 00:49:34 on dist-test-slave-86rs"
server_key: "73e2eca86679923d0485099368e86fcd"
server_key_iv: "08ada63e7d043bf873febc2696f79abc"
server_key_version: "kuduclusterkey@0"
I20260927 00:49:34.124703 12876 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.005s sys 0.000s
I20260927 00:49:34.142311 12892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:34.144408 12876 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:49:34.144683 12876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "1e4417c04271412f83c037bda3a3883c"
format_stamp: "Formatted at 2026-09-27 00:49:34 on dist-test-slave-86rs"
server_key: "73e2eca86679923d0485099368e86fcd"
server_key_iv: "08ada63e7d043bf873febc2696f79abc"
server_key_version: "kuduclusterkey@0"
I20260927 00:49:34.146523 12876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:49:34.211737 12876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:49:34.215842 12876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:49:34.216154 12876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:49:34.217090 12876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:49:34.218300 12876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:49:34.218417 12876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:34.218525 12876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:49:34.218595 12876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:34.233536 12876 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41281
I20260927 00:49:34.235018 12876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:49:34.244607 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 12876
I20260927 00:49:34.244740 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:49:34.245110 392 external_mini_cluster.cc:1428] Setting key 59c8c6824c53b8172eaf23b942c245e7
I20260927 00:49:34.245061 13005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41281 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 27 00:49:34 dist-test-slave-86rs krb5kdc[12180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:49:34.284494 12874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:34.253629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37401 (local address 127.0.98.62:33621)
0927 00:49:34.255361 (+ 1732us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:34.255366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:34.264940 (+ 9574us) server_negotiation.cc:407] Connection header received
0927 00:49:34.266531 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:34.266536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:34.266602 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:34.266660 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:34.270889 (+ 4229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:34.271742 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:34.273614 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:34.273983 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:34.274063 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:34.278116 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:34.278121 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:34.278124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:34.278159 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:34.279977 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:34.282555 (+ 2578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:34.282559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:34.282561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:34.282660 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:34.283080 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:34.283084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:34.283085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:34.283320 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:34.283398 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:34.284110 (+ 712us) server_negotiation.cc:299] Negotiation successful
0927 00:49:34.284284 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1622}
I20260927 00:49:34.288877 13009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:34.254229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33621 (local address 127.0.98.2:37401)
0927 00:49:34.264697 (+ 10468us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:34.264799 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:34.266077 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:34.267153 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:34.267164 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:34.267655 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:49:34.270654 (+ 2999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:34.270679 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:34.271933 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:34.271936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:34.273435 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:34.273449 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:34.274295 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:34.274300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:34.274527 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:34.275331 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:49:34.275353 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:49:34.277880 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:49:34.280131 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:34.280137 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:34.280157 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:34.282403 (+ 2246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:34.282771 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:34.282774 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:34.282776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:34.282969 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:34.283474 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:49:34.283481 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:49:34.283828 (+ 347us) client_negotiation.cc:769] Sending connection context
0927 00:49:34.284971 (+ 1143us) client_negotiation.cc:240] Negotiation successful
0927 00:49:34.285341 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10122,"thread_start_us":183,"threads_started":1}
I20260927 00:49:34.290936 13006 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33621
I20260927 00:49:34.291239 13006 heartbeater.cc:464] Registering TS with master...
I20260927 00:49:34.291870 13006 heartbeater.cc:511] Master 127.0.98.62:33621 requested a full tablet report, sending...
I20260927 00:49:34.293275 12656 ts_manager.cc:194] Registered new tserver with Master: 1e4417c04271412f83c037bda3a3883c (127.0.98.2:41281)
I20260927 00:49:34.294037 12656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:49:34.324717 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51685/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33621
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:49:34 dist-test-slave-86rs krb5kdc[12180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 00:49:34.504858 13011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:49:34.505332 13011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:49:34.505456 13011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:49:34.511582 13011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:49:34.511881 13011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:49:34.512029 13011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:49:34.520215 13011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:51685/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:49:34.521704 13011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:49:34.523083 13011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:49:34.535784 13019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:34.536381 13016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:34.536448 13011 server_base.cc:1034] running on GCE node
W20260927 00:49:34.535853 13017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:34.537145 13011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:49:34.538046 13011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:49:34.539256 13011 hybrid_clock.cc:648] HybridClock initialized: now 1790470174539230 us; error 42 us; skew 500 ppm
Sep 27 00:49:34 dist-test-slave-86rs krb5kdc[12180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:49:34.543208 13011 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:49:34.544469 13011 webserver.cc:460] Webserver started at http://127.0.98.3:36879/ using document root <none> and password file <none>
I20260927 00:49:34.545207 13011 fs_manager.cc:359] Metadata directory not provided
I20260927 00:49:34.545290 13011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:49:34.545521 13011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:49:34 dist-test-slave-86rs krb5kdc[12180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:49:34.564388 13011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/data/instance:
uuid: "6dcf375e992a477f8c1f6b0c938e4bf4"
format_stamp: "Formatted at 2026-09-27 00:49:34 on dist-test-slave-86rs"
server_key: "3365f33fd9ae652568a70a2c699564d6"
server_key_iv: "e79dcd9e25a30f8115c2b0d14c496fc0"
server_key_version: "kuduclusterkey@0"
I20260927 00:49:34.565322 13011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6dcf375e992a477f8c1f6b0c938e4bf4"
format_stamp: "Formatted at 2026-09-27 00:49:34 on dist-test-slave-86rs"
server_key: "3365f33fd9ae652568a70a2c699564d6"
server_key_iv: "e79dcd9e25a30f8115c2b0d14c496fc0"
server_key_version: "kuduclusterkey@0"
I20260927 00:49:34.570809 13011 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 00:49:34.603626 13026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:34.605221 13011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:49:34.605383 13011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "6dcf375e992a477f8c1f6b0c938e4bf4"
format_stamp: "Formatted at 2026-09-27 00:49:34 on dist-test-slave-86rs"
server_key: "3365f33fd9ae652568a70a2c699564d6"
server_key_iv: "e79dcd9e25a30f8115c2b0d14c496fc0"
server_key_version: "kuduclusterkey@0"
I20260927 00:49:34.605554 13011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:49:34.633731 13011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:49:34.637882 13011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:49:34.638168 13011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:49:34.639016 13011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:49:34.640203 13011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:49:34.640362 13011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:34.640488 13011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:49:34.640532 13011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:34.652035 13011 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44841
I20260927 00:49:34.653332 13011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:49:34.655793 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 13011
I20260927 00:49:34.655922 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:49:34.656384 392 external_mini_cluster.cc:1428] Setting key 194fd915f3844f0f428d200643bf4efc
I20260927 00:49:34.664151 13140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44841 every 8 connection(s)
Sep 27 00:49:34 dist-test-slave-86rs krb5kdc[12180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 00:49:34 dist-test-slave-86rs krb5kdc[12180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:49:34.683790 12874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:34.657654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59691 (local address 127.0.98.62:33621)
0927 00:49:34.657801 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:34.657806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:34.659184 (+ 1378us) server_negotiation.cc:407] Connection header received
0927 00:49:34.660513 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:34.660518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:34.660573 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:34.660621 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:34.662364 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:34.663170 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:34.664913 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:34.665254 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:34.665336 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:34.669301 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:34.669306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:34.669309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:34.669342 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:34.671123 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:34.679783 (+ 8660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:34.679788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:34.679790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:34.679954 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:34.682247 (+ 2293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:34.682252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:34.682254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:34.682540 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:34.682650 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:34.683460 (+ 810us) server_negotiation.cc:299] Negotiation successful
0927 00:49:34.683617 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:49:34.685918 13143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:34.656367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33621 (local address 127.0.98.3:59691)
0927 00:49:34.659012 (+ 2645us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:34.659086 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:34.660217 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:34.661076 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:34.661087 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:34.661459 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:49:34.662184 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:34.662206 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:34.663370 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:34.663373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:34.664777 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:34.664790 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:34.665571 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:34.665575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:34.665787 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:34.666495 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:49:34.666518 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:49:34.669005 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:49:34.671636 (+ 2631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:34.671644 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:34.671668 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:34.677076 (+ 5408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:34.681299 (+ 4223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:34.681307 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:34.681310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:34.682092 (+ 782us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:34.682733 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:49:34.682742 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:49:34.683160 (+ 418us) client_negotiation.cc:769] Sending connection context
0927 00:49:34.684324 (+ 1164us) client_negotiation.cc:240] Negotiation successful
0927 00:49:34.684738 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":176,"threads_started":1}
I20260927 00:49:34.687636 13141 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33621
I20260927 00:49:34.687898 13141 heartbeater.cc:464] Registering TS with master...
I20260927 00:49:34.688565 13141 heartbeater.cc:511] Master 127.0.98.62:33621 requested a full tablet report, sending...
I20260927 00:49:34.689956 12656 ts_manager.cc:194] Registered new tserver with Master: 6dcf375e992a477f8c1f6b0c938e4bf4 (127.0.98.3:44841)
I20260927 00:49:34.690768 12656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59691
I20260927 00:49:34.695480 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:49:34.696324 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 12737
I20260927 00:49:34.704535 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 12876
I20260927 00:49:34.713105 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 13011
I20260927 00:49:34.720561 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 12624
I20260927 00:49:34.729540 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33621
--webserver_interface=127.0.98.62
--webserver_port=39583
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33621
--ranger_config_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51685/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51685/kms with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:49:34.858412 13145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:49:34.858714 13145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:49:34.858757 13145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:49:34.862993 13145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:49:34.863065 13145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:49:34.863085 13145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:49:34.863101 13145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:49:34.863152 13145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:49:34.868927 13145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51685/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33621
--ranger_config_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:49:34.870100 13145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:49:34.871191 13145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:49:34.880059 13150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:34.880035 13153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:34.880101 13151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:34.880543 13145 server_base.cc:1034] running on GCE node
I20260927 00:49:34.881181 13145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:49:34.882331 13145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:49:34.883529 13145 hybrid_clock.cc:648] HybridClock initialized: now 1790470174883515 us; error 35 us; skew 500 ppm
Sep 27 00:49:34 dist-test-slave-86rs krb5kdc[12180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:49:34.886422 13145 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:49:34.887439 13145 webserver.cc:460] Webserver started at http://127.0.98.62:39583/ using document root <none> and password file <none>
I20260927 00:49:34.888042 13145 fs_manager.cc:359] Metadata directory not provided
I20260927 00:49:34.888141 13145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 00:49:34 dist-test-slave-86rs krb5kdc[12180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:49:34.939383 13145 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:49:34.942304 13161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:34.944027 13145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:49:34.944180 13145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "b2d0a9d6b7ea4b5a86384eb6585a1644"
format_stamp: "Formatted at 2026-09-27 00:49:32 on dist-test-slave-86rs"
server_key: "441697c4cd4553f177d363615394ed58"
server_key_iv: "40a8c7ef5fa5d8cd6353f159bf88b52e"
server_key_version: "kuduclusterkey@0"
I20260927 00:49:34.944970 13145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:49:34.965179 13145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:49:34.967780 13145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:49:34.967938 13145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:49:34.975265 13145 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33621
I20260927 00:49:34.975270 13213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33621 every 8 connection(s)
I20260927 00:49:34.976415 13145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:49:34.978467 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 13145
I20260927 00:49:34.978932 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51685/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45583
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33621
--webserver_port=35257
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51685/kms with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:49:34.981115 13214 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:49:34.982826 13214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644: Bootstrap starting.
I20260927 00:49:34.999578 13214 log.cc:824] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644: Log is configured to *not* fsync() on all Append() calls
I20260927 00:49:35.006603 13214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:49:35.007038 13214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644: Bootstrap complete.
I20260927 00:49:35.011283 13214 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d0a9d6b7ea4b5a86384eb6585a1644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33621 } }
I20260927 00:49:35.011734 13214 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d0a9d6b7ea4b5a86384eb6585a1644, State: Initialized, Role: FOLLOWER
I20260927 00:49:35.012351 13214 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d0a9d6b7ea4b5a86384eb6585a1644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33621 } }
I20260927 00:49:35.012503 13214 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:49:35.012573 13214 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:49:35.012668 13214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:49:35.014693 13214 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d0a9d6b7ea4b5a86384eb6585a1644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33621 } }
I20260927 00:49:35.015070 13214 leader_election.cc:302] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2d0a9d6b7ea4b5a86384eb6585a1644; no voters:
I20260927 00:49:35.015404 13214 leader_election.cc:288] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 00:49:35.015580 13218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 00:49:35.015841 13218 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [term 2 LEADER]: Becoming Leader. State: Replica: b2d0a9d6b7ea4b5a86384eb6585a1644, State: Running, Role: LEADER
I20260927 00:49:35.016279 13218 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d0a9d6b7ea4b5a86384eb6585a1644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33621 } }
I20260927 00:49:35.016769 13214 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:49:35.017346 13218 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2d0a9d6b7ea4b5a86384eb6585a1644. Latest consensus state: current_term: 2 leader_uuid: "b2d0a9d6b7ea4b5a86384eb6585a1644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d0a9d6b7ea4b5a86384eb6585a1644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33621 } } }
I20260927 00:49:35.017550 13218 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [sys.catalog]: This master's current role is: LEADER
I20260927 00:49:35.017769 13219 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b2d0a9d6b7ea4b5a86384eb6585a1644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d0a9d6b7ea4b5a86384eb6585a1644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33621 } } }
I20260927 00:49:35.017882 13219 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644 [sys.catalog]: This master's current role is: LEADER
I20260927 00:49:35.017966 13224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:49:35.021309 13224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:49:35.021886 13214 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 00:49:35.021996 13224 catalog_manager.cc:1259] Loaded cluster ID: a85fbc09fdfd49329179f36eb4c2b391
I20260927 00:49:35.022051 13224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:49:35.024612 13224 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:49:35.025444 13224 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644: Loaded TSK: 0
I20260927 00:49:35.026686 13224 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:49:35.128930 13216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:49:35.129302 13216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:49:35.129370 13216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:49:35.135361 13216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:49:35.135492 13216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:49:35.135610 13216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:49:35.142854 13216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51685/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35257
--tserver_master_addrs=127.0.98.62:33621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:49:35.144393 13216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:49:35.145661 13216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:49:35.156093 13252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:35.156673 13254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:35.156782 13216 server_base.cc:1034] running on GCE node
W20260927 00:49:35.157238 13251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:35.157801 13216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:49:35.158743 13216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:49:35.159932 13216 hybrid_clock.cc:648] HybridClock initialized: now 1790470175159899 us; error 63 us; skew 500 ppm
Sep 27 00:49:35 dist-test-slave-86rs krb5kdc[12180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:49:35.163205 13216 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:49:35.164525 13216 webserver.cc:460] Webserver started at http://127.0.98.1:35257/ using document root <none> and password file <none>
I20260927 00:49:35.165238 13216 fs_manager.cc:359] Metadata directory not provided
I20260927 00:49:35.165387 13216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 00:49:35 dist-test-slave-86rs krb5kdc[12180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:49:35.193614 13216 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.002s sys 0.001s
I20260927 00:49:35.197861 13262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:35.199658 13216 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:49:35.199905 13216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "e5cce0c6ed734684ae4fba815a1c2d19"
format_stamp: "Formatted at 2026-09-27 00:49:33 on dist-test-slave-86rs"
server_key: "4485da15a05be4b3fe72e0991c0a49ac"
server_key_iv: "635ef8849682614810c268de55e78315"
server_key_version: "kuduclusterkey@0"
I20260927 00:49:35.200861 13216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:49:35.218569 13216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:49:35.223104 13216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:49:35.223419 13216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:49:35.224270 13216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:49:35.225435 13216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:49:35.225540 13216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:35.225648 13216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:49:35.225705 13216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:35.237377 13216 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45583
I20260927 00:49:35.238879 13216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:49:35.240316 13375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45583 every 8 connection(s)
I20260927 00:49:35.240542 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 13216
I20260927 00:49:35.241025 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51685/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41281
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33621
--webserver_port=46681
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51685/kms with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:49:35 dist-test-slave-86rs krb5kdc[12180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:49:35.283811 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:35.243166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33621 (local address 127.0.98.1:34203)
0927 00:49:35.248671 (+ 5505us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:35.248756 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:35.250002 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:35.262317 (+ 12315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:35.262328 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:35.262754 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:49:35.263473 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:35.263496 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:35.265984 (+ 2488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:35.265988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:35.267440 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:35.267455 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:35.274669 (+ 7214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:35.274679 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:35.274896 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:35.275677 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:49:35.275701 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:49:35.277488 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:49:35.280117 (+ 2629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:35.280123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:35.280137 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:35.280530 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:35.280977 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:35.280980 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:35.280982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:35.281163 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:35.281969 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:49:35.281976 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:49:35.282292 (+ 316us) client_negotiation.cc:769] Sending connection context
0927 00:49:35.282478 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 00:49:35.282813 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5190,"thread_start_us":160,"threads_started":1}
I20260927 00:49:35.286140 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:35.244833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34203 (local address 127.0.98.62:33621)
0927 00:49:35.250646 (+ 5813us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:35.250675 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:35.250726 (+ 51us) server_negotiation.cc:407] Connection header received
0927 00:49:35.251050 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:35.251066 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:35.261075 (+ 10009us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:35.261616 (+ 541us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:35.264115 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:35.265772 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:35.267645 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:35.274151 (+ 6506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:35.274374 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:35.277733 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:35.277739 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:35.277753 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:35.277785 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:35.279941 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:35.280692 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:35.280699 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:35.280705 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:35.280850 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:35.281280 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:35.281283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:35.281285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:35.281773 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:35.281891 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:35.284840 (+ 2949us) server_negotiation.cc:299] Negotiation successful
0927 00:49:35.285179 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5635,"thread_start_us":131,"threads_started":1}
I20260927 00:49:35.289371 13376 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33621
I20260927 00:49:35.289642 13376 heartbeater.cc:464] Registering TS with master...
I20260927 00:49:35.290230 13376 heartbeater.cc:511] Master 127.0.98.62:33621 requested a full tablet report, sending...
W20260927 00:49:35.292675 13376 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33621 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 00:49:35.292802 13376 heartbeater.cc:464] Registering TS with master...
I20260927 00:49:35.293025 13376 heartbeater.cc:511] Master 127.0.98.62:33621 requested a full tablet report, sending...
I20260927 00:49:35.294167 13376 heartbeater.cc:464] Registering TS with master...
I20260927 00:49:35.294353 13376 heartbeater.cc:511] Master 127.0.98.62:33621 requested a full tablet report, sending...
W20260927 00:49:35.295251 13376 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:49:35.411612 13378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:49:35.412070 13378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:49:35.412153 13378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:49:35.417243 13378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:49:35.417332 13378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:49:35.417436 13378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:49:35.423904 13378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51685/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46681
--tserver_master_addrs=127.0.98.62:33621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:49:35.425247 13378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:49:35.426331 13378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:49:35.436659 13385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:35.436597 13388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:35.436712 13386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:35.437130 13378 server_base.cc:1034] running on GCE node
I20260927 00:49:35.437676 13378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:49:35.438578 13378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:49:35.439769 13378 hybrid_clock.cc:648] HybridClock initialized: now 1790470175439769 us; error 52 us; skew 500 ppm
Sep 27 00:49:35 dist-test-slave-86rs krb5kdc[12180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:49:35.442824 13378 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:49:35.443727 13378 webserver.cc:460] Webserver started at http://127.0.98.2:46681/ using document root <none> and password file <none>
I20260927 00:49:35.444375 13378 fs_manager.cc:359] Metadata directory not provided
I20260927 00:49:35.444480 13378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 00:49:35 dist-test-slave-86rs krb5kdc[12180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:49:35.462075 13378 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260927 00:49:35.469245 13396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:35.471110 13378 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.002s
I20260927 00:49:35.471268 13378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "1e4417c04271412f83c037bda3a3883c"
format_stamp: "Formatted at 2026-09-27 00:49:34 on dist-test-slave-86rs"
server_key: "73e2eca86679923d0485099368e86fcd"
server_key_iv: "08ada63e7d043bf873febc2696f79abc"
server_key_version: "kuduclusterkey@0"
I20260927 00:49:35.472079 13378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:49:35.489022 13378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:49:35.493198 13378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:49:35.493433 13378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:49:35.494249 13378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:49:35.495450 13378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:49:35.495565 13378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260927 00:49:35.495661 13378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:49:35.495718 13378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:35.508003 13378 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41281
I20260927 00:49:35.508101 13509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41281 every 8 connection(s)
I20260927 00:49:35.509613 13378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:49:35.519960 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 13378
I20260927 00:49:35.520473 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51685/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44841
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33621
--webserver_port=36879
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51685/kms with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:49:35 dist-test-slave-86rs krb5kdc[12180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:49:35.531620 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:35.511807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32919 (local address 127.0.98.62:33621)
0927 00:49:35.511947 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:35.511952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:35.513027 (+ 1075us) server_negotiation.cc:407] Connection header received
0927 00:49:35.514055 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:35.514059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:35.514117 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:35.514199 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:35.515535 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:35.516115 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:35.517471 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:35.517720 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:35.517852 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:35.526611 (+ 8759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:35.526616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:35.526619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:35.526658 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:35.528602 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:35.529355 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:35.529360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:35.529361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:35.529462 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:35.529965 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:35.529969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:35.529971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:35.530373 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:35.530461 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:35.531268 (+ 807us) server_negotiation.cc:299] Negotiation successful
0927 00:49:35.531417 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:49:35.533965 13512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:35.512369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33621 (local address 127.0.98.2:32919)
0927 00:49:35.512872 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:35.512926 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:35.513828 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:35.514388 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:35.514395 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:35.514678 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:49:35.515342 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:35.515363 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:35.516305 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:35.516308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:35.517343 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:35.517353 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:35.518181 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:35.518184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:35.518408 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:35.523419 (+ 5011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:49:35.523454 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:49:35.526311 (+ 2857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:49:35.528766 (+ 2455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:35.528775 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:35.528795 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:35.529214 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:35.529608 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:35.529611 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:35.529614 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:35.529838 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:35.530531 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:49:35.530539 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:49:35.530949 (+ 410us) client_negotiation.cc:769] Sending connection context
0927 00:49:35.532349 (+ 1400us) client_negotiation.cc:240] Negotiation successful
0927 00:49:35.532766 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":152,"threads_started":1}
I20260927 00:49:35.536784 13510 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33621
I20260927 00:49:35.537101 13510 heartbeater.cc:464] Registering TS with master...
I20260927 00:49:35.537703 13510 heartbeater.cc:511] Master 127.0.98.62:33621 requested a full tablet report, sending...
W20260927 00:49:35.539561 13510 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33621 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 00:49:35.539688 13510 heartbeater.cc:464] Registering TS with master...
I20260927 00:49:35.539865 13510 heartbeater.cc:511] Master 127.0.98.62:33621 requested a full tablet report, sending...
I20260927 00:49:35.540817 13510 heartbeater.cc:464] Registering TS with master...
I20260927 00:49:35.540988 13510 heartbeater.cc:511] Master 127.0.98.62:33621 requested a full tablet report, sending...
W20260927 00:49:35.541836 13510 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:49:35.683518 13513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:49:35.684021 13513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:49:35.684106 13513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:49:35.690634 13513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:49:35.690799 13513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:49:35.690928 13513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:49:35.699483 13513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:51685/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36879
--tserver_master_addrs=127.0.98.62:33621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:49:35.701040 13513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:49:35.702445 13513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:49:35.715317 13519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:35.715415 13522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:35.715318 13520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:35.716499 13513 server_base.cc:1034] running on GCE node
I20260927 00:49:35.717103 13513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:49:35.718068 13513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:49:35.719224 13513 hybrid_clock.cc:648] HybridClock initialized: now 1790470175719201 us; error 43 us; skew 500 ppm
Sep 27 00:49:35 dist-test-slave-86rs krb5kdc[12180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:49:35.722278 13513 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:49:35.723251 13513 webserver.cc:460] Webserver started at http://127.0.98.3:36879/ using document root <none> and password file <none>
I20260927 00:49:35.723856 13513 fs_manager.cc:359] Metadata directory not provided
I20260927 00:49:35.723973 13513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 00:49:35 dist-test-slave-86rs krb5kdc[12180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:49:35.749404 13513 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:49:35.752959 13530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:35.754665 13513 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:49:35.754818 13513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "6dcf375e992a477f8c1f6b0c938e4bf4"
format_stamp: "Formatted at 2026-09-27 00:49:34 on dist-test-slave-86rs"
server_key: "3365f33fd9ae652568a70a2c699564d6"
server_key_iv: "e79dcd9e25a30f8115c2b0d14c496fc0"
server_key_version: "kuduclusterkey@0"
I20260927 00:49:35.755620 13513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:49:35.810066 13513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:49:35.813797 13513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:49:35.814015 13513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:49:35.814787 13513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:49:35.815881 13513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:49:35.815951 13513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:35.816010 13513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:49:35.816043 13513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:35.827611 13513 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44841
I20260927 00:49:35.827675 13643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44841 every 8 connection(s)
I20260927 00:49:35.829078 13513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:49:35.833133 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 13513
Sep 27 00:49:35 dist-test-slave-86rs krb5kdc[12180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:49:35.844949 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:35.833785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60028 (local address 127.0.98.62:33621)
0927 00:49:35.835060 (+ 1275us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:35.835068 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:35.835079 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:49:35.835125 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:35.835128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:35.835174 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:35.835258 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:35.836264 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:35.837696 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:35.838798 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:35.839090 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:35.839192 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:35.841108 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:35.841112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:35.841116 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:35.841147 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:35.842989 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:35.843501 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:35.843505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:35.843507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:35.843609 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:35.843897 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:35.843900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:35.843902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:35.844136 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:35.844372 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:35.844601 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 00:49:35.844737 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":1205,"thread_start_us":107,"threads_started":1}
I20260927 00:49:35.846452 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:35.831455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41779 (local address 127.0.98.62:33621)
0927 00:49:35.831581 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:35.831586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:35.832739 (+ 1153us) server_negotiation.cc:407] Connection header received
0927 00:49:35.834519 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:35.834522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:35.834574 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:35.834669 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:35.836281 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:35.836853 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:35.838484 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:35.838870 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:35.838994 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:35.841957 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:35.841963 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:35.841971 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:35.842040 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:35.843610 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:35.844364 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:35.844367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:35.844369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:35.844451 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:35.844893 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:35.844896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:35.844898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:35.845128 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:35.845205 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:35.846176 (+ 971us) server_negotiation.cc:299] Negotiation successful
0927 00:49:35.846294 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":46}
I20260927 00:49:35.847491 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:35.831878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33621 (local address 127.0.98.3:41779)
0927 00:49:35.832378 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:35.832433 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:35.834304 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:35.834988 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:35.834997 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:35.835363 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:49:35.836103 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:35.836124 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:35.836985 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:35.836989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:35.838335 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:35.838345 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:35.839055 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:35.839059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:35.839286 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:35.840014 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:49:35.840043 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:49:35.841780 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:49:35.843912 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:35.843918 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:35.843933 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:35.844224 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:35.844563 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:35.844566 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:35.844568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:35.844773 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:35.845268 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:49:35.845275 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:49:35.845852 (+ 577us) client_negotiation.cc:769] Sending connection context
0927 00:49:35.846116 (+ 264us) client_negotiation.cc:240] Negotiation successful
0927 00:49:35.846467 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":134,"threads_started":1}
I20260927 00:49:35.848728 13644 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33621
I20260927 00:49:35.849028 13644 heartbeater.cc:464] Registering TS with master...
I20260927 00:49:35.849709 13644 heartbeater.cc:511] Master 127.0.98.62:33621 requested a full tablet report, sending...
I20260927 00:49:35.851300 13178 ts_manager.cc:194] Registered new tserver with Master: 6dcf375e992a477f8c1f6b0c938e4bf4 (127.0.98.3:44841)
I20260927 00:49:35.852984 13178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41779
Sep 27 00:49:36 dist-test-slave-86rs krb5kdc[12180](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790470176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:49:36.295527 13376 heartbeater.cc:464] Registering TS with master...
I20260927 00:49:36.295876 13376 heartbeater.cc:511] Master 127.0.98.62:33621 requested a full tablet report, sending...
I20260927 00:49:36.296977 13178 ts_manager.cc:194] Registered new tserver with Master: e5cce0c6ed734684ae4fba815a1c2d19 (127.0.98.1:45583)
I20260927 00:49:36.297847 13178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34203
I20260927 00:49:36.542111 13510 heartbeater.cc:464] Registering TS with master...
I20260927 00:49:36.542454 13510 heartbeater.cc:511] Master 127.0.98.62:33621 requested a full tablet report, sending...
I20260927 00:49:36.543382 13178 ts_manager.cc:194] Registered new tserver with Master: 1e4417c04271412f83c037bda3a3883c (127.0.98.2:41281)
I20260927 00:49:36.544173 13178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:32919
I20260927 00:49:36.551308 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:49:36.567000 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:36.553229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60044 (local address 127.0.98.62:33621)
0927 00:49:36.556511 (+ 3282us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:36.556516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:36.556637 (+ 121us) server_negotiation.cc:407] Connection header received
0927 00:49:36.556794 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:36.556798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:36.556856 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:36.556964 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:36.557907 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:36.558771 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:36.560048 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:36.560441 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:36.560560 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:36.562621 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:36.562625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:36.562628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:36.562660 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:36.564602 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:36.565141 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:36.565145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:36.565147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:36.565248 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:36.565576 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:36.565579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:36.565581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:36.565821 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:36.565964 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:36.566657 (+ 693us) server_negotiation.cc:299] Negotiation successful
0927 00:49:36.566803 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3176,"thread_start_us":71,"threads_started":1}
I20260927 00:49:36.573530 13178 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:49:36.584200 13178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 27 00:49:36 dist-test-slave-86rs krb5kdc[12180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:49:36.611436 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:36.598913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45583 (local address 127.0.0.1:42240)
0927 00:49:36.599659 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:36.599675 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:36.599765 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:36.601434 (+ 1669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:36.601442 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:36.601480 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:49:36.601651 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:36.601659 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:36.603087 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:36.603090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:36.604813 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:36.604825 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:36.610778 (+ 5953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:36.610781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:36.610931 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:36.610974 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 00:49:36.611112 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 00:49:36.611266 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":124,"threads_started":1}
I20260927 00:49:36.612210 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:36.599741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42240 (local address 127.0.98.1:45583)
0927 00:49:36.600097 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:36.600106 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:36.600135 (+ 29us) server_negotiation.cc:407] Connection header received
0927 00:49:36.600214 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:36.600245 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:36.601246 (+ 1001us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:36.601413 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:49:36.601795 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:36.602939 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:36.604960 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:36.610418 (+ 5458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:36.610533 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:36.610701 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:36.611935 (+ 1234us) server_negotiation.cc:299] Negotiation successful
0927 00:49:36.612066 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":129,"threads_started":1}
I20260927 00:49:36.616082 13291 tablet_service.cc:1467] Processing CreateTablet for tablet 0e73944139c0432d975c1dc18aaee176 (DEFAULT_TABLE table=test-table [id=f2dfcc0d0e99455684512035ddcb74f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:49:36.617668 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e73944139c0432d975c1dc18aaee176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:49:36.622660 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:36.599191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44841 (local address 127.0.0.1:55856)
0927 00:49:36.599951 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:36.599964 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:36.600046 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:36.601796 (+ 1750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:36.601801 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:36.601820 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:49:36.601912 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:36.601917 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:36.605534 (+ 3617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:36.605538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:36.606751 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:36.606765 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:36.622118 (+ 15353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:36.622124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:36.622263 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:36.622300 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 00:49:36.622387 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 00:49:36.622492 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":79,"threads_started":1}
I20260927 00:49:36.623453 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:36.599377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55856 (local address 127.0.98.3:44841)
0927 00:49:36.599689 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:36.599701 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:36.600948 (+ 1247us) server_negotiation.cc:407] Connection header received
0927 00:49:36.601038 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:36.601044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:36.601243 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:36.601412 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:49:36.604262 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:36.605381 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:36.608304 (+ 2923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:36.609274 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:36.609394 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:36.623125 (+ 13731us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:36.623190 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 00:49:36.623308 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":81,"threads_started":1}
I20260927 00:49:36.626080 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:36.597109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41281 (local address 127.0.0.1:35338)
0927 00:49:36.597553 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:36.598149 (+ 596us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:36.598290 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:36.602195 (+ 3905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:36.602198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:36.602216 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:49:36.602303 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:36.602309 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:36.622961 (+ 20652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:36.622965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:36.624160 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:36.624175 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:36.625652 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:36.625656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:36.625759 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:36.625786 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 00:49:36.625839 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 00:49:36.625921 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":158,"threads_started":1}
I20260927 00:49:36.627076 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:36.597255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35338 (local address 127.0.98.2:41281)
0927 00:49:36.601707 (+ 4452us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:36.601712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:36.601756 (+ 44us) server_negotiation.cc:407] Connection header received
0927 00:49:36.601829 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:36.601837 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:36.602033 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:36.602447 (+ 414us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:49:36.602619 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:36.607220 (+ 4601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:36.624597 (+ 17377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:36.625480 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:36.626606 (+ 1126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:36.626775 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:36.626834 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 00:49:36.626935 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4351,"thread_start_us":72,"threads_started":1}
I20260927 00:49:36.627168 13578 tablet_service.cc:1467] Processing CreateTablet for tablet 0e73944139c0432d975c1dc18aaee176 (DEFAULT_TABLE table=test-table [id=f2dfcc0d0e99455684512035ddcb74f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:49:36.628643 13578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e73944139c0432d975c1dc18aaee176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:49:36.630018 13434 tablet_service.cc:1467] Processing CreateTablet for tablet 0e73944139c0432d975c1dc18aaee176 (DEFAULT_TABLE table=test-table [id=f2dfcc0d0e99455684512035ddcb74f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:49:36.630903 13677 tablet_bootstrap.cc:492] T 0e73944139c0432d975c1dc18aaee176 P e5cce0c6ed734684ae4fba815a1c2d19: Bootstrap starting.
I20260927 00:49:36.631067 13434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e73944139c0432d975c1dc18aaee176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:49:36.633472 13677 tablet_bootstrap.cc:654] T 0e73944139c0432d975c1dc18aaee176 P e5cce0c6ed734684ae4fba815a1c2d19: Neither blocks nor log segments found. Creating new log.
I20260927 00:49:36.634384 13677 log.cc:824] T 0e73944139c0432d975c1dc18aaee176 P e5cce0c6ed734684ae4fba815a1c2d19: Log is configured to *not* fsync() on all Append() calls
I20260927 00:49:36.638249 13677 tablet_bootstrap.cc:492] T 0e73944139c0432d975c1dc18aaee176 P e5cce0c6ed734684ae4fba815a1c2d19: No bootstrap required, opened a new log
I20260927 00:49:36.638598 13677 ts_tablet_manager.cc:1397] T 0e73944139c0432d975c1dc18aaee176 P e5cce0c6ed734684ae4fba815a1c2d19: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 00:49:36.645068 13677 raft_consensus.cc:348] T 0e73944139c0432d975c1dc18aaee176 P e5cce0c6ed734684ae4fba815a1c2d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4417c04271412f83c037bda3a3883c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "e5cce0c6ed734684ae4fba815a1c2d19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45583 } } peers { permanent_uuid: "6dcf375e992a477f8c1f6b0c938e4bf4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44841 } }
I20260927 00:49:36.645648 13677 raft_consensus.cc:374] T 0e73944139c0432d975c1dc18aaee176 P e5cce0c6ed734684ae4fba815a1c2d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:49:36.645772 13677 raft_consensus.cc:729] T 0e73944139c0432d975c1dc18aaee176 P e5cce0c6ed734684ae4fba815a1c2d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5cce0c6ed734684ae4fba815a1c2d19, State: Initialized, Role: FOLLOWER
I20260927 00:49:36.646435 13677 consensus_queue.cc:260] T 0e73944139c0432d975c1dc18aaee176 P e5cce0c6ed734684ae4fba815a1c2d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4417c04271412f83c037bda3a3883c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "e5cce0c6ed734684ae4fba815a1c2d19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45583 } } peers { permanent_uuid: "6dcf375e992a477f8c1f6b0c938e4bf4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44841 } }
I20260927 00:49:36.646693 13679 tablet_bootstrap.cc:492] T 0e73944139c0432d975c1dc18aaee176 P 6dcf375e992a477f8c1f6b0c938e4bf4: Bootstrap starting.
I20260927 00:49:36.647878 13677 ts_tablet_manager.cc:1428] T 0e73944139c0432d975c1dc18aaee176 P e5cce0c6ed734684ae4fba815a1c2d19: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260927 00:49:36.648384 13376 heartbeater.cc:503] Master 127.0.98.62:33621 was elected leader, sending a full tablet report...
I20260927 00:49:36.653774 13679 tablet_bootstrap.cc:654] T 0e73944139c0432d975c1dc18aaee176 P 6dcf375e992a477f8c1f6b0c938e4bf4: Neither blocks nor log segments found. Creating new log.
I20260927 00:49:36.654971 13679 log.cc:824] T 0e73944139c0432d975c1dc18aaee176 P 6dcf375e992a477f8c1f6b0c938e4bf4: Log is configured to *not* fsync() on all Append() calls
I20260927 00:49:36.655615 13680 tablet_bootstrap.cc:492] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c: Bootstrap starting.
I20260927 00:49:36.658952 13680 tablet_bootstrap.cc:654] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c: Neither blocks nor log segments found. Creating new log.
I20260927 00:49:36.659957 13680 log.cc:824] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:49:36.664762 13679 tablet_bootstrap.cc:492] T 0e73944139c0432d975c1dc18aaee176 P 6dcf375e992a477f8c1f6b0c938e4bf4: No bootstrap required, opened a new log
I20260927 00:49:36.665170 13679 ts_tablet_manager.cc:1397] T 0e73944139c0432d975c1dc18aaee176 P 6dcf375e992a477f8c1f6b0c938e4bf4: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.000s
I20260927 00:49:36.670262 13679 raft_consensus.cc:348] T 0e73944139c0432d975c1dc18aaee176 P 6dcf375e992a477f8c1f6b0c938e4bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4417c04271412f83c037bda3a3883c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "e5cce0c6ed734684ae4fba815a1c2d19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45583 } } peers { permanent_uuid: "6dcf375e992a477f8c1f6b0c938e4bf4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44841 } }
I20260927 00:49:36.670650 13679 raft_consensus.cc:374] T 0e73944139c0432d975c1dc18aaee176 P 6dcf375e992a477f8c1f6b0c938e4bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:49:36.677659 13680 tablet_bootstrap.cc:492] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c: No bootstrap required, opened a new log
I20260927 00:49:36.677932 13680 ts_tablet_manager.cc:1397] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.000s
I20260927 00:49:36.682601 13680 raft_consensus.cc:348] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4417c04271412f83c037bda3a3883c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "e5cce0c6ed734684ae4fba815a1c2d19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45583 } } peers { permanent_uuid: "6dcf375e992a477f8c1f6b0c938e4bf4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44841 } }
I20260927 00:49:36.684343 13679 raft_consensus.cc:729] T 0e73944139c0432d975c1dc18aaee176 P 6dcf375e992a477f8c1f6b0c938e4bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dcf375e992a477f8c1f6b0c938e4bf4, State: Initialized, Role: FOLLOWER
I20260927 00:49:36.685038 13679 consensus_queue.cc:260] T 0e73944139c0432d975c1dc18aaee176 P 6dcf375e992a477f8c1f6b0c938e4bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4417c04271412f83c037bda3a3883c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "e5cce0c6ed734684ae4fba815a1c2d19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45583 } } peers { permanent_uuid: "6dcf375e992a477f8c1f6b0c938e4bf4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44841 } }
I20260927 00:49:36.686388 13679 ts_tablet_manager.cc:1428] T 0e73944139c0432d975c1dc18aaee176 P 6dcf375e992a477f8c1f6b0c938e4bf4: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20260927 00:49:36.686827 13644 heartbeater.cc:503] Master 127.0.98.62:33621 was elected leader, sending a full tablet report...
I20260927 00:49:36.686830 13680 raft_consensus.cc:374] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:49:36.686909 13680 raft_consensus.cc:729] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e4417c04271412f83c037bda3a3883c, State: Initialized, Role: FOLLOWER
I20260927 00:49:36.687544 13680 consensus_queue.cc:260] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4417c04271412f83c037bda3a3883c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "e5cce0c6ed734684ae4fba815a1c2d19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45583 } } peers { permanent_uuid: "6dcf375e992a477f8c1f6b0c938e4bf4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44841 } }
I20260927 00:49:36.688823 13680 ts_tablet_manager.cc:1428] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260927 00:49:36.689271 13510 heartbeater.cc:503] Master 127.0.98.62:33621 was elected leader, sending a full tablet report...
W20260927 00:49:36.742129 13377 tablet.cc:2369] T 0e73944139c0432d975c1dc18aaee176 P e5cce0c6ed734684ae4fba815a1c2d19: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:49:36.762423 13511 tablet.cc:2369] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:49:36.810756 13685 raft_consensus.cc:482] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:49:36.811033 13685 raft_consensus.cc:504] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4417c04271412f83c037bda3a3883c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "e5cce0c6ed734684ae4fba815a1c2d19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45583 } } peers { permanent_uuid: "6dcf375e992a477f8c1f6b0c938e4bf4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44841 } }
I20260927 00:49:36.812566 13685 leader_election.cc:288] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5cce0c6ed734684ae4fba815a1c2d19 (127.0.98.1:45583), 6dcf375e992a477f8c1f6b0c938e4bf4 (127.0.98.3:44841)
I20260927 00:49:36.819295 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:36.813523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42829 (local address 127.0.98.3:44841)
0927 00:49:36.814065 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:36.814071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:36.814534 (+ 463us) server_negotiation.cc:407] Connection header received
0927 00:49:36.814750 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:36.814754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:36.814806 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:36.814857 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:49:36.815345 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:36.816113 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:36.817849 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:36.818502 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:36.818601 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:36.818681 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:36.819053 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 00:49:36.819144 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443}
I20260927 00:49:36.820012 13686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:36.813014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44841 (local address 127.0.98.2:42829)
0927 00:49:36.814426 (+ 1412us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:36.814454 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:36.814625 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:36.815021 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:36.815024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:36.815045 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:49:36.815199 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:36.815207 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:36.816309 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:36.816312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:36.817701 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:36.817712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:36.818784 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:36.818788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:36.818895 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:36.818919 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:49:36.819772 (+ 853us) client_negotiation.cc:240] Negotiation successful
0927 00:49:36.819860 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1338,"thread_start_us":155,"threads_started":1}
I20260927 00:49:36.820904 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e73944139c0432d975c1dc18aaee176" candidate_uuid: "1e4417c04271412f83c037bda3a3883c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dcf375e992a477f8c1f6b0c938e4bf4" is_pre_election: true
I20260927 00:49:36.821313 13580 raft_consensus.cc:2454] T 0e73944139c0432d975c1dc18aaee176 P 6dcf375e992a477f8c1f6b0c938e4bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e4417c04271412f83c037bda3a3883c in term 0.
I20260927 00:49:36.821980 13399 leader_election.cc:302] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e4417c04271412f83c037bda3a3883c, 6dcf375e992a477f8c1f6b0c938e4bf4; no voters:
I20260927 00:49:36.822281 13685 raft_consensus.cc:2781] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:49:36.822371 13685 raft_consensus.cc:482] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:49:36.822414 13685 raft_consensus.cc:3037] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:49:36.823814 13685 raft_consensus.cc:504] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4417c04271412f83c037bda3a3883c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "e5cce0c6ed734684ae4fba815a1c2d19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45583 } } peers { permanent_uuid: "6dcf375e992a477f8c1f6b0c938e4bf4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44841 } }
I20260927 00:49:36.824862 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e73944139c0432d975c1dc18aaee176" candidate_uuid: "1e4417c04271412f83c037bda3a3883c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dcf375e992a477f8c1f6b0c938e4bf4"
I20260927 00:49:36.824997 13580 raft_consensus.cc:3037] T 0e73944139c0432d975c1dc18aaee176 P 6dcf375e992a477f8c1f6b0c938e4bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:49:36.826391 13580 raft_consensus.cc:2454] T 0e73944139c0432d975c1dc18aaee176 P 6dcf375e992a477f8c1f6b0c938e4bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e4417c04271412f83c037bda3a3883c in term 1.
I20260927 00:49:36.826843 13399 leader_election.cc:302] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e4417c04271412f83c037bda3a3883c, 6dcf375e992a477f8c1f6b0c938e4bf4; no voters:
I20260927 00:49:36.827184 13685 leader_election.cc:288] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c [CANDIDATE]: Term 1 election: Requested vote from peers e5cce0c6ed734684ae4fba815a1c2d19 (127.0.98.1:45583), 6dcf375e992a477f8c1f6b0c938e4bf4 (127.0.98.3:44841)
I20260927 00:49:36.827288 13685 raft_consensus.cc:2781] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:49:36.827381 13685 raft_consensus.cc:686] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c [term 1 LEADER]: Becoming Leader. State: Replica: 1e4417c04271412f83c037bda3a3883c, State: Running, Role: LEADER
I20260927 00:49:36.827772 13685 consensus_queue.cc:237] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4417c04271412f83c037bda3a3883c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } } peers { permanent_uuid: "e5cce0c6ed734684ae4fba815a1c2d19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45583 } } peers { permanent_uuid: "6dcf375e992a477f8c1f6b0c938e4bf4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44841 } }
W20260927 00:49:36.830865 13645 tablet.cc:2369] T 0e73944139c0432d975c1dc18aaee176 P 6dcf375e992a477f8c1f6b0c938e4bf4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:49:36.839345 13178 catalog_manager.cc:5495] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e4417c04271412f83c037bda3a3883c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1e4417c04271412f83c037bda3a3883c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4417c04271412f83c037bda3a3883c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5cce0c6ed734684ae4fba815a1c2d19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dcf375e992a477f8c1f6b0c938e4bf4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44841 } health_report { overall_health: UNKNOWN } } }
I20260927 00:49:36.845238 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:36.814222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53761 (local address 127.0.98.1:45583)
0927 00:49:36.814376 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:36.814381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:36.820776 (+ 6395us) server_negotiation.cc:407] Connection header received
0927 00:49:36.820992 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:36.820995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:36.821055 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:36.821105 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:49:36.821477 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:36.822159 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:36.843733 (+ 21574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:36.844466 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:36.844559 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:36.844631 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:36.845006 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 00:49:36.845102 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:49:36.845871 13686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:36.813843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45583 (local address 127.0.98.2:53761)
0927 00:49:36.820696 (+ 6853us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:36.820711 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:36.820857 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:36.821239 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:36.821242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:36.821260 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:49:36.821334 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:36.821343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:36.840392 (+ 19049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:36.840395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:36.843529 (+ 3134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:36.843547 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:36.844740 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:36.844744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:36.844847 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:36.844872 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:49:36.845645 (+ 773us) client_negotiation.cc:240] Negotiation successful
0927 00:49:36.845738 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6791,"thread_start_us":102,"threads_started":1}
I20260927 00:49:36.846751 13328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e73944139c0432d975c1dc18aaee176" candidate_uuid: "1e4417c04271412f83c037bda3a3883c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5cce0c6ed734684ae4fba815a1c2d19" is_pre_election: true
I20260927 00:49:36.847117 13328 raft_consensus.cc:2454] T 0e73944139c0432d975c1dc18aaee176 P e5cce0c6ed734684ae4fba815a1c2d19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e4417c04271412f83c037bda3a3883c in term 0.
I20260927 00:49:36.847450 13322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e73944139c0432d975c1dc18aaee176" candidate_uuid: "1e4417c04271412f83c037bda3a3883c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5cce0c6ed734684ae4fba815a1c2d19"
I20260927 00:49:36.847551 13322 raft_consensus.cc:3037] T 0e73944139c0432d975c1dc18aaee176 P e5cce0c6ed734684ae4fba815a1c2d19 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:49:36.848882 13322 raft_consensus.cc:2454] T 0e73944139c0432d975c1dc18aaee176 P e5cce0c6ed734684ae4fba815a1c2d19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e4417c04271412f83c037bda3a3883c in term 1.
Sep 27 00:49:36 dist-test-slave-86rs krb5kdc[12180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:49:36.915483 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:36.910428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35348 (local address 127.0.98.2:41281)
0927 00:49:36.910592 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:36.910598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:36.910707 (+ 109us) server_negotiation.cc:407] Connection header received
0927 00:49:36.910931 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:36.910935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:36.911015 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:36.911145 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:49:36.911414 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:36.911943 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:36.913226 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:36.913596 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:36.913696 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:36.914120 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:49:36.914743 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:49:36.914867 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:36.915244 (+ 377us) server_negotiation.cc:299] Negotiation successful
0927 00:49:36.915315 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:49:36.924037 13580 raft_consensus.cc:1261] T 0e73944139c0432d975c1dc18aaee176 P 6dcf375e992a477f8c1f6b0c938e4bf4 [term 1 FOLLOWER]: Refusing update from remote peer 1e4417c04271412f83c037bda3a3883c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:49:36.924774 13322 raft_consensus.cc:1261] T 0e73944139c0432d975c1dc18aaee176 P e5cce0c6ed734684ae4fba815a1c2d19 [term 1 FOLLOWER]: Refusing update from remote peer 1e4417c04271412f83c037bda3a3883c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:49:36.925829 13688 consensus_queue.cc:1035] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5cce0c6ed734684ae4fba815a1c2d19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:49:36.929003 13688 consensus_queue.cc:1035] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dcf375e992a477f8c1f6b0c938e4bf4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:49:36.951550 13692 mvcc.cc:205] Tried to move back new op lower bound from 7333765844669972480 to 7333765844321689600. Current Snapshot: MvccSnapshot[applied={T|T < 7333765844669972480}]
I20260927 00:49:36.954007 13695 mvcc.cc:205] Tried to move back new op lower bound from 7333765844669972480 to 7333765844321689600. Current Snapshot: MvccSnapshot[applied={T|T < 7333765844669972480}]
I20260927 00:49:36.957481 13693 mvcc.cc:205] Tried to move back new op lower bound from 7333765844669972480 to 7333765844321689600. Current Snapshot: MvccSnapshot[applied={T|T < 7333765844669972480}]
I20260927 00:49:36.971422 13178 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:49:36.971643 13178 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:49:36.975078 13178 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2d0a9d6b7ea4b5a86384eb6585a1644: Sending DeleteTablet for 3 replicas of tablet 0e73944139c0432d975c1dc18aaee176
I20260927 00:49:36.976068 13434 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e73944139c0432d975c1dc18aaee176 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:49:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35338
I20260927 00:49:36.977246 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e73944139c0432d975c1dc18aaee176 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:49:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42240
I20260927 00:49:36.978266 13578 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e73944139c0432d975c1dc18aaee176 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:49:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55856
I20260927 00:49:36.979879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 13216
I20260927 00:49:36.986794 13703 tablet_replica.cc:330] T 0e73944139c0432d975c1dc18aaee176 P 6dcf375e992a477f8c1f6b0c938e4bf4: stopping tablet replica
I20260927 00:49:36.987293 13703 raft_consensus.cc:2229] T 0e73944139c0432d975c1dc18aaee176 P 6dcf375e992a477f8c1f6b0c938e4bf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:49:36.987778 13703 raft_consensus.cc:2258] T 0e73944139c0432d975c1dc18aaee176 P 6dcf375e992a477f8c1f6b0c938e4bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:49:36.989816 13703 ts_tablet_manager.cc:1905] T 0e73944139c0432d975c1dc18aaee176 P 6dcf375e992a477f8c1f6b0c938e4bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:49:36.990211 13701 tablet_replica.cc:330] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c: stopping tablet replica
I20260927 00:49:36.990639 13701 raft_consensus.cc:2229] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:49:36.991441 13701 raft_consensus.cc:2258] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:49:36.992746 13703 ts_tablet_manager.cc:1918] T 0e73944139c0432d975c1dc18aaee176 P 6dcf375e992a477f8c1f6b0c938e4bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:49:36.992854 13703 log.cc:1196] T 0e73944139c0432d975c1dc18aaee176 P 6dcf375e992a477f8c1f6b0c938e4bf4: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-2/wal/wals/0e73944139c0432d975c1dc18aaee176
I20260927 00:49:36.993199 13703 ts_tablet_manager.cc:1939] T 0e73944139c0432d975c1dc18aaee176 P 6dcf375e992a477f8c1f6b0c938e4bf4: Deleting consensus metadata
I20260927 00:49:36.994415 13701 ts_tablet_manager.cc:1905] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:49:36.994621 13164 catalog_manager.cc:4841] TS 6dcf375e992a477f8c1f6b0c938e4bf4 (127.0.98.3:44841): tablet 0e73944139c0432d975c1dc18aaee176 (table test-table [id=f2dfcc0d0e99455684512035ddcb74f3]) successfully deleted
I20260927 00:49:37.007305 13701 ts_tablet_manager.cc:1918] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:49:37.007558 13701 log.cc:1196] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790470006085796-392-0/minicluster-data/ts-1/wal/wals/0e73944139c0432d975c1dc18aaee176
I20260927 00:49:37.008093 13701 ts_tablet_manager.cc:1939] T 0e73944139c0432d975c1dc18aaee176 P 1e4417c04271412f83c037bda3a3883c: Deleting consensus metadata
I20260927 00:49:37.009925 13163 catalog_manager.cc:4841] TS 1e4417c04271412f83c037bda3a3883c (127.0.98.2:41281): tablet 0e73944139c0432d975c1dc18aaee176 (table test-table [id=f2dfcc0d0e99455684512035ddcb74f3]) successfully deleted
W20260927 00:49:37.012439 13165 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45583 (error 108)
I20260927 00:49:37.012538 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 13378
I20260927 00:49:37.022109 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 13513
I20260927 00:49:37.025331 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:37.023973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45583 (local address 127.0.0.1:42242)
0927 00:49:37.024203 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:37.025220 (+ 1017us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260927 00:49:37.025599 13165 catalog_manager.cc:4584] TS e5cce0c6ed734684ae4fba815a1c2d19 (127.0.98.1:45583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0e73944139c0432d975c1dc18aaee176: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45583: connect: Connection refused (error 111)
I20260927 00:49:37.029673 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 13145
I20260927 00:49:37.036916 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 00:49:37.382205 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 00:49:37.382364 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 00:49:37.728094 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 00:49:37.728 UTC [12210] LOG: received smart shutdown request
2026-09-27 00:49:37.732 UTC [12210] LOG: background worker "logical replication launcher" (PID 12219) exited with exit code 1
2026-09-27 00:49:37.732 UTC [12214] LOG: shutting down
2026-09-27 00:49:37.922 UTC [12210] LOG: database system is shut down
2026-09-27T00:49:37Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51205 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:49:38 dist-test-slave-86rs krb5kdc[13713](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:49:38 dist-test-slave-86rs krb5kdc[13713](info): set up 2 sockets
Sep 27 00:49:38 dist-test-slave-86rs krb5kdc[13713](info): commencing operation
krb5kdc: starting...
W20260927 00:49:39.100528 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:49:39 dist-test-slave-86rs krb5kdc[13713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:49:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:49:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:49:39.227900 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42023
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42023,127.0.98.61:39221,127.0.98.60:33419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:49:39.344683 13726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:49:39.345041 13726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:49:39.345131 13726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:49:39.349334 13726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:49:39.349440 13726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:49:39.349479 13726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:49:39.349517 13726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:49:39.349553 13726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:49:39.355016 13726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42023,127.0.98.61:39221,127.0.98.60:33419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:49:39.356297 13726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:49:39.357400 13726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:49:39.366528 13734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:39.366489 13731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:39.366478 13732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:39.366788 13726 server_base.cc:1034] running on GCE node
I20260927 00:49:39.367555 13726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:49:39.368961 13726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:49:39.370162 13726 hybrid_clock.cc:648] HybridClock initialized: now 1790470179370130 us; error 55 us; skew 500 ppm
Sep 27 00:49:39 dist-test-slave-86rs krb5kdc[13713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:49:39.373143 13726 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:49:39.374114 13726 webserver.cc:460] Webserver started at http://127.0.98.62:46559/ using document root <none> and password file <none>
I20260927 00:49:39.374655 13726 fs_manager.cc:359] Metadata directory not provided
I20260927 00:49:39.374712 13726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:49:39.374881 13726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:49:39.377127 13726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "9c6a771b03054ac3bf6960ab8e51f72e"
format_stamp: "Formatted at 2026-09-27 00:49:39 on dist-test-slave-86rs"
server_key: "83f21356a01699139d477e97923f5842"
server_key_iv: "1fe3ae0178f62506c328ad0067cf784f"
server_key_version: "encryptionkey@0"
I20260927 00:49:39.377720 13726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "9c6a771b03054ac3bf6960ab8e51f72e"
format_stamp: "Formatted at 2026-09-27 00:49:39 on dist-test-slave-86rs"
server_key: "83f21356a01699139d477e97923f5842"
server_key_iv: "1fe3ae0178f62506c328ad0067cf784f"
server_key_version: "encryptionkey@0"
I20260927 00:49:39.381898 13726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:49:39.384871 13741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:39.386343 13726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:49:39.386521 13726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "9c6a771b03054ac3bf6960ab8e51f72e"
format_stamp: "Formatted at 2026-09-27 00:49:39 on dist-test-slave-86rs"
server_key: "83f21356a01699139d477e97923f5842"
server_key_iv: "1fe3ae0178f62506c328ad0067cf784f"
server_key_version: "encryptionkey@0"
I20260927 00:49:39.386638 13726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:49:39.414230 13726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:49:39.418926 13726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:49:39.419148 13726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:49:39.427379 13726 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42023
I20260927 00:49:39.427387 13793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42023 every 8 connection(s)
I20260927 00:49:39.428503 13726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:49:39.431411 13794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:49:39.435230 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 13726
I20260927 00:49:39.435360 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:49:39.435669 392 external_mini_cluster.cc:1428] Setting key a9d8397c8a3cb339b76d54bdb8157268
I20260927 00:49:39.435524 13794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42023 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 00:49:39 dist-test-slave-86rs krb5kdc[13713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:49:39.453320 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.438605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42023 (local address 127.0.0.1:48338)
0927 00:49:39.439293 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:39.439331 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:39.440322 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:39.441036 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:39.441042 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:39.441506 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:49:39.441772 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:39.441785 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.442927 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.442930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:39.443671 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:39.443679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.444449 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.444453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:39.444571 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:39.445242 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:49:39.445259 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:49:39.447645 (+ 2386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:49:39.449868 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:39.449881 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:39.449896 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:39.450257 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:39.450637 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:39.450643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:39.450648 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:39.450813 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:39.451580 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:49:39.451588 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:49:39.451789 (+ 201us) client_negotiation.cc:769] Sending connection context
0927 00:49:39.452015 (+ 226us) client_negotiation.cc:240] Negotiation successful
0927 00:49:39.452547 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":117,"spinlock_wait_cycles":14080,"thread_start_us":185,"threads_started":1}
I20260927 00:49:39.453323 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.438604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48338 (local address 127.0.98.62:42023)
0927 00:49:39.439090 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:39.439106 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:39.439452 (+ 346us) server_negotiation.cc:407] Connection header received
0927 00:49:39.440645 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:39.440650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:39.440844 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:39.441049 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:39.441941 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.442801 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.443794 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.444292 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.444463 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:39.447836 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:39.447847 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:39.447866 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:39.447900 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:39.449738 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.450391 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.450397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.450398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.450517 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.450948 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.450954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.450955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.451410 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:39.451567 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:39.452093 (+ 526us) server_negotiation.cc:299] Negotiation successful
0927 00:49:39.452440 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":329,"thread_start_us":171,"threads_started":1}
I20260927 00:49:39.455847 13794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39221 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:49:39.456739 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.456451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39221 (local address 127.0.0.1:43042)
0927 00:49:39.456580 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:39.456663 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 00:49:39.457020 13743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39221: connect: Connection refused (error 111)
I20260927 00:49:39.458006 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.457761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39221 (local address 127.0.0.1:43056)
0927 00:49:39.457882 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:39.457932 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 00:49:39.458179 13794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39221: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39221: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 00:49:39.465155 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39221
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42023,127.0.98.61:39221,127.0.98.60:33419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:49:39.507537 13794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39221 } attempt: 1
I20260927 00:49:39.508654 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.508171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39221 (local address 127.0.0.1:43058)
0927 00:49:39.508418 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:39.508513 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260927 00:49:39.509500 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.509258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39221 (local address 127.0.0.1:43064)
0927 00:49:39.509378 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:39.509435 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 00:49:39.509732 13794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39221: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39221: connect: Connection refused (error 111)
I20260927 00:49:39.577958 13794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39221 } attempt: 2
I20260927 00:49:39.578982 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.578605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39221 (local address 127.0.0.1:43080)
0927 00:49:39.578798 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:39.578877 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 00:49:39.579874 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.579624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39221 (local address 127.0.0.1:43090)
0927 00:49:39.579748 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:39.579804 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 00:49:39.580040 13794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39221: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39221: connect: Connection refused (error 111)
W20260927 00:49:39.581063 13801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:49:39.581385 13801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:49:39.581461 13801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:49:39.585728 13801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:49:39.585815 13801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:49:39.585865 13801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:49:39.585901 13801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:49:39.585932 13801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:49:39.591708 13801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42023,127.0.98.61:39221,127.0.98.60:33419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:49:39.593009 13801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:49:39.594112 13801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:49:39.603183 13809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:39.603183 13807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:39.603210 13806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:39.603915 13801 server_base.cc:1034] running on GCE node
I20260927 00:49:39.604601 13801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:49:39.605801 13801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:49:39.607030 13801 hybrid_clock.cc:648] HybridClock initialized: now 1790470179607009 us; error 34 us; skew 500 ppm
Sep 27 00:49:39 dist-test-slave-86rs krb5kdc[13713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:49:39.610049 13801 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 00:49:39.611055 13801 webserver.cc:460] Webserver started at http://127.0.98.61:42901/ using document root <none> and password file <none>
I20260927 00:49:39.611685 13801 fs_manager.cc:359] Metadata directory not provided
I20260927 00:49:39.611780 13801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:49:39.612028 13801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:49:39.614292 13801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-1/data/instance:
uuid: "4fb5f67ae39e4c0a852804f4aacb2d33"
format_stamp: "Formatted at 2026-09-27 00:49:39 on dist-test-slave-86rs"
server_key: "006fbfb64058131d95053d1183fdda17"
server_key_iv: "cdbad652ccb721cd9d032c7ecc8c0342"
server_key_version: "encryptionkey@0"
I20260927 00:49:39.614953 13801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-1/wal/instance:
uuid: "4fb5f67ae39e4c0a852804f4aacb2d33"
format_stamp: "Formatted at 2026-09-27 00:49:39 on dist-test-slave-86rs"
server_key: "006fbfb64058131d95053d1183fdda17"
server_key_iv: "cdbad652ccb721cd9d032c7ecc8c0342"
server_key_version: "encryptionkey@0"
I20260927 00:49:39.619117 13801 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 00:49:39.621865 13816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:39.623236 13801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:49:39.623405 13801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-1/wal
uuid: "4fb5f67ae39e4c0a852804f4aacb2d33"
format_stamp: "Formatted at 2026-09-27 00:49:39 on dist-test-slave-86rs"
server_key: "006fbfb64058131d95053d1183fdda17"
server_key_iv: "cdbad652ccb721cd9d032c7ecc8c0342"
server_key_version: "encryptionkey@0"
I20260927 00:49:39.623535 13801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:49:39.649233 13801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:49:39.652930 13801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:49:39.653174 13801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:49:39.660754 13801 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39221
I20260927 00:49:39.660781 13868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39221 every 8 connection(s)
I20260927 00:49:39.661813 13801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-1/data/info.pb
I20260927 00:49:39.662616 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 13801
I20260927 00:49:39.662757 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-1/wal/instance
I20260927 00:49:39.663097 392 external_mini_cluster.cc:1428] Setting key 2a45959c6a723937bf2f173ba9d7f03d
I20260927 00:49:39.665627 13869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:49:39.670063 13869 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42023 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 00:49:39.671224 13794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39221 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 00:49:39 dist-test-slave-86rs krb5kdc[13713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 00:49:39 dist-test-slave-86rs krb5kdc[13713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:49:39.685398 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.672188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48346 (local address 127.0.98.62:42023)
0927 00:49:39.672313 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:39.672318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:39.673174 (+ 856us) server_negotiation.cc:407] Connection header received
0927 00:49:39.674189 (+ 1015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:39.674192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:39.674255 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:39.674354 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:39.675582 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.676384 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.678025 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.678381 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.678478 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:39.681482 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:39.681487 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:39.681491 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:39.681520 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:39.683030 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.683660 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.683663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.683664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.683786 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.684187 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.684190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.684192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.684460 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:39.684562 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:39.685045 (+ 483us) server_negotiation.cc:299] Negotiation successful
0927 00:49:39.685209 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":59}
I20260927 00:49:39.685911 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.671792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39221 (local address 127.0.0.1:43106)
0927 00:49:39.672022 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:39.672040 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:39.672153 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:39.674453 (+ 2300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:39.674459 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:39.674685 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:49:39.674848 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:39.674858 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.677085 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.677088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:39.677614 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:39.677621 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.678391 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.678395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:39.678542 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:39.679178 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:49:39.679206 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:49:39.680906 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:49:39.683365 (+ 2459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:39.683368 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:39.683370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:39.683627 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:39.683998 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:39.684001 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:39.684003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:39.684087 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:39.684833 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:49:39.684836 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:49:39.684943 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 00:49:39.685049 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 00:49:39.685285 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":63}
I20260927 00:49:39.686381 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.672426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43106 (local address 127.0.98.61:39221)
0927 00:49:39.672913 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:39.672928 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:39.672965 (+ 37us) server_negotiation.cc:407] Connection header received
0927 00:49:39.673897 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:39.673925 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:39.674252 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:39.675029 (+ 777us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:39.675184 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.676639 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.677744 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.678245 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.678428 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:39.681104 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:39.681114 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:39.681129 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:39.681163 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:39.683254 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.683752 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.683759 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.683761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.683899 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.684187 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.684190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.684192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.684726 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:39.684829 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:39.685159 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 00:49:39.685362 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":293,"thread_start_us":151,"threads_started":1}
I20260927 00:49:39.686959 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.672426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42023 (local address 127.0.0.1:48346)
0927 00:49:39.673001 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:39.673061 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:39.673923 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:39.674411 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:39.674420 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:39.674784 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:49:39.675406 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:39.675426 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.676532 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.676539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:39.677890 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:39.677905 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.678661 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.678663 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:39.678788 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:39.679341 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:49:39.679358 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:49:39.681316 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:49:39.683158 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:39.683169 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:39.683182 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:39.683533 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:39.683881 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:39.683884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:39.683886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:39.684073 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:39.684600 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:49:39.684605 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:49:39.684748 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 00:49:39.684961 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 00:49:39.686762 (+ 1801us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":1475,"spinlock_wait_cycles":2304,"thread_start_us":209,"threads_started":1}
I20260927 00:49:39.688197 13869 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39221 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:49:39.688967 13794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33419 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:49:39.689813 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.689553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33419 (local address 127.0.0.1:36136)
0927 00:49:39.689679 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:39.689738 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 00:49:39.690694 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.690368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33419 (local address 127.0.0.1:36152)
0927 00:49:39.690567 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:39.690626 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260927 00:49:39.690896 13794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33419: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33419: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 00:49:39.692873 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33419
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42023,127.0.98.61:39221,127.0.98.60:33419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:49:39 dist-test-slave-86rs krb5kdc[13713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:49:39.697696 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.688811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39221 (local address 127.0.0.1:43108)
0927 00:49:39.688920 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:39.688936 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:39.689022 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:39.689661 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:39.689664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:39.689867 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:49:39.689992 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:39.689999 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.691070 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.691073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:39.691805 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:39.691818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.692462 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.692465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:39.692573 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:39.693106 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:49:39.693118 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:49:39.694359 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:49:39.696110 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:39.696115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:39.696117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:39.696379 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:39.696668 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:39.696670 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:39.696672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:39.696754 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:39.697137 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:49:39.697140 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:49:39.697233 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 00:49:39.697333 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 00:49:39.697486 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 00:49:39.697696 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.689352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43108 (local address 127.0.98.61:39221)
0927 00:49:39.689471 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:39.689475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:39.689490 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:49:39.689524 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:39.689526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:39.689560 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:39.689652 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:39.690160 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.690939 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.691947 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.692334 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.692441 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:39.694509 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:39.694512 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:39.694514 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:39.694534 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:39.695979 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.696477 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.696482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.696492 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.696568 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.696837 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.696843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.696845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.697037 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:39.697138 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:39.697387 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 00:49:39.697553 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":51}
I20260927 00:49:39.699155 13869 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33419 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:49:39.699958 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.699683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33419 (local address 127.0.0.1:36166)
0927 00:49:39.699808 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:39.699892 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 00:49:39.700618 13818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33419: connect: Connection refused (error 111)
I20260927 00:49:39.701565 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.701352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33419 (local address 127.0.0.1:36178)
0927 00:49:39.701459 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:39.701510 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 00:49:39.701728 13869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33419: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33419: connect: Connection refused (error 111)
I20260927 00:49:39.722090 13794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33419 } attempt: 1
I20260927 00:49:39.723099 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.722769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33419 (local address 127.0.0.1:36188)
0927 00:49:39.722936 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:39.723012 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 00:49:39.723869 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.723646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33419 (local address 127.0.0.1:36192)
0927 00:49:39.723759 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:39.723812 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 00:49:39.723999 13794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33419: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33419: connect: Connection refused (error 111)
I20260927 00:49:39.751029 13869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33419 } attempt: 1
I20260927 00:49:39.752099 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.751707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33419 (local address 127.0.0.1:36208)
0927 00:49:39.751923 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:39.751997 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260927 00:49:39.753038 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.752757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33419 (local address 127.0.0.1:36214)
0927 00:49:39.752911 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:39.752969 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260927 00:49:39.753171 13869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33419: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33419: connect: Connection refused (error 111)
I20260927 00:49:39.799190 13794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33419 } attempt: 2
I20260927 00:49:39.800169 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.799854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33419 (local address 127.0.0.1:36228)
0927 00:49:39.800012 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:39.800084 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 00:49:39.801033 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.800802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33419 (local address 127.0.0.1:36240)
0927 00:49:39.800920 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:39.800972 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 00:49:39.801168 13794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33419: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33419: connect: Connection refused (error 111)
W20260927 00:49:39.814272 13875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:49:39.814630 13875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:49:39.814728 13875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:49:39.819151 13875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:49:39.819257 13875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:49:39.819296 13875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:49:39.819355 13875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:49:39.819411 13875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:49:39.821348 13869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33419 } attempt: 2
I20260927 00:49:39.822322 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.821993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33419 (local address 127.0.0.1:36242)
0927 00:49:39.822165 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:39.822236 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 00:49:39.823161 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.822892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33419 (local address 127.0.0.1:36254)
0927 00:49:39.823039 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:39.823094 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 00:49:39.823302 13869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33419: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33419: connect: Connection refused (error 111)
I20260927 00:49:39.825800 13875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42023,127.0.98.61:39221,127.0.98.60:33419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:49:39.827101 13875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:49:39.828348 13875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:49:39.837263 13884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:39.837265 13881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:39.837266 13882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:39.837574 13875 server_base.cc:1034] running on GCE node
I20260927 00:49:39.838294 13875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:49:39.839601 13875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:49:39.840837 13875 hybrid_clock.cc:648] HybridClock initialized: now 1790470179840818 us; error 35 us; skew 500 ppm
Sep 27 00:49:39 dist-test-slave-86rs krb5kdc[13713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:49:39.844162 13875 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 00:49:39.845320 13875 webserver.cc:460] Webserver started at http://127.0.98.60:39743/ using document root <none> and password file <none>
I20260927 00:49:39.845969 13875 fs_manager.cc:359] Metadata directory not provided
I20260927 00:49:39.846084 13875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:49:39.846354 13875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:49:39.848896 13875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-2/data/instance:
uuid: "d83c3266fdd645a88a9c61c87d1e7387"
format_stamp: "Formatted at 2026-09-27 00:49:39 on dist-test-slave-86rs"
server_key: "43bcb6794bcbc72ca76f0adb2379eeb6"
server_key_iv: "0deae72bb02754464c0fa84f5d9b22be"
server_key_version: "encryptionkey@0"
I20260927 00:49:39.849565 13875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-2/wal/instance:
uuid: "d83c3266fdd645a88a9c61c87d1e7387"
format_stamp: "Formatted at 2026-09-27 00:49:39 on dist-test-slave-86rs"
server_key: "43bcb6794bcbc72ca76f0adb2379eeb6"
server_key_iv: "0deae72bb02754464c0fa84f5d9b22be"
server_key_version: "encryptionkey@0"
I20260927 00:49:39.853899 13875 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:49:39.856705 13891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:39.858101 13875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:49:39.858294 13875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-2/wal
uuid: "d83c3266fdd645a88a9c61c87d1e7387"
format_stamp: "Formatted at 2026-09-27 00:49:39 on dist-test-slave-86rs"
server_key: "43bcb6794bcbc72ca76f0adb2379eeb6"
server_key_iv: "0deae72bb02754464c0fa84f5d9b22be"
server_key_version: "encryptionkey@0"
I20260927 00:49:39.858436 13875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:49:39.872799 13875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:49:39.876037 13875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:49:39.876327 13875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:49:39.883872 13875 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33419
I20260927 00:49:39.883886 13943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33419 every 8 connection(s)
I20260927 00:49:39.885118 13875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-2/data/info.pb
I20260927 00:49:39.888144 13944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:49:39.890007 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 13875
I20260927 00:49:39.890172 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790470006085796-392-0/minicluster-data/master-2/wal/instance
I20260927 00:49:39.890498 392 external_mini_cluster.cc:1428] Setting key 69969c5361e1ed068d4520f10953c49c
I20260927 00:49:39.892907 13944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42023 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:49:39.900316 13794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33419 } attempt: 3
Sep 27 00:49:39 dist-test-slave-86rs krb5kdc[13713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 00:49:39 dist-test-slave-86rs krb5kdc[13713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 00:49:39 dist-test-slave-86rs krb5kdc[13713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:49:39.906399 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.895484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48348 (local address 127.0.98.62:42023)
0927 00:49:39.895618 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:39.895623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:39.895911 (+ 288us) server_negotiation.cc:407] Connection header received
0927 00:49:39.896040 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:39.896045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:39.896110 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:39.896178 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:39.897041 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.897527 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.898451 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.898835 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.898948 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:39.901711 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:39.901714 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:39.901717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:39.901741 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:39.904389 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.904939 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.904944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.904946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.905034 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.905358 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.905362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.905363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.905576 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:39.905649 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:39.906113 (+ 464us) server_negotiation.cc:299] Negotiation successful
0927 00:49:39.906240 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58,"spinlock_wait_cycles":768}
I20260927 00:49:39.907220 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.895347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42023 (local address 127.0.0.1:48348)
0927 00:49:39.895771 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:39.895821 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:39.895945 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:39.896286 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:39.896292 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:39.896760 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:49:39.896913 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:39.896918 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.897661 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.897665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:39.898318 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:39.898327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.898999 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.899003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:39.899122 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:39.899681 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:49:39.899697 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:49:39.901552 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:49:39.904493 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:39.904504 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:39.904510 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:39.904838 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:39.905151 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:39.905154 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:39.905156 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:39.905252 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:39.905724 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:49:39.905729 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:49:39.905858 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 00:49:39.906544 (+ 686us) client_negotiation.cc:240] Negotiation successful
0927 00:49:39.906749 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":1,"thread_start_us":114,"threads_started":1}
I20260927 00:49:39.907235 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.892080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36260 (local address 127.0.98.60:33419)
0927 00:49:39.892592 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:39.892611 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:39.892657 (+ 46us) server_negotiation.cc:407] Connection header received
0927 00:49:39.893464 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:39.893492 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:39.893970 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:39.894367 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:39.895149 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.896682 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.897557 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.898045 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.898233 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:39.901270 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:39.901277 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:39.901289 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:39.901311 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:39.903237 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.903696 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.903705 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.903716 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.903883 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.904234 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.904257 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.904259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.904897 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:39.905108 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:39.905432 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 00:49:39.905962 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":344,"thread_start_us":186,"threads_started":1}
I20260927 00:49:39.909407 13944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39221 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:49:39.910931 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.900815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33419 (local address 127.0.0.1:36272)
0927 00:49:39.901011 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:39.901027 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:39.901122 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:39.901650 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:39.901654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:39.901926 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:49:39.902024 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:39.902029 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.903021 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.903025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:39.903553 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:39.903562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.904176 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.904183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:39.904312 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:39.904779 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:49:39.904794 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:49:39.906294 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:49:39.909003 (+ 2709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:39.909006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:39.909008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:39.909252 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:39.909571 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:39.909575 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:39.909576 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:39.909667 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:39.910378 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:49:39.910381 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:49:39.910479 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 00:49:39.910596 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 00:49:39.910739 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260927 00:49:39.910972 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.900891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36272 (local address 127.0.98.60:33419)
0927 00:49:39.901365 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:39.901374 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:39.901386 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:49:39.901418 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:39.901420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:39.901469 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:39.901606 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:39.902143 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.902911 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.903674 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.904045 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.904164 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:39.906440 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:39.906444 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:39.906447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:39.906472 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:39.908802 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.909352 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.909355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.909357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.909446 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.909786 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.909788 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.909790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.910004 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:39.910388 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:39.910665 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 00:49:39.910785 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":127,"threads_started":1}
I20260927 00:49:39.914450 13869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33419 } attempt: 3
Sep 27 00:49:39 dist-test-slave-86rs krb5kdc[13713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:49:39.915043 13794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c6a771b03054ac3bf6960ab8e51f72e: Bootstrap starting.
Sep 27 00:49:39 dist-test-slave-86rs krb5kdc[13713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:49:39.918435 13794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c6a771b03054ac3bf6960ab8e51f72e: Neither blocks nor log segments found. Creating new log.
I20260927 00:49:39.919477 13794 log.cc:824] T 00000000000000000000000000000000 P 9c6a771b03054ac3bf6960ab8e51f72e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:49:39.920117 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.910032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43110 (local address 127.0.98.61:39221)
0927 00:49:39.910142 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:39.910147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:39.910328 (+ 181us) server_negotiation.cc:407] Connection header received
0927 00:49:39.910478 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:39.910481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:39.910530 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:39.910634 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:39.911153 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.911796 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.912896 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.913200 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.913273 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:39.915533 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:39.915538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:39.915540 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:39.915609 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:39.917354 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.918323 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.918327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.918329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.918427 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.918844 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.918850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.918855 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.919068 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:39.919137 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:39.919484 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 00:49:39.919925 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 00:49:39.920684 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.909949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39221 (local address 127.0.0.1:43110)
0927 00:49:39.910258 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:39.910271 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:39.910390 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:39.910749 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:39.910754 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:39.910902 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:49:39.911008 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:39.911017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.911923 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.911926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:39.912756 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:39.912770 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.913420 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.913423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:39.913531 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:39.913916 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:49:39.913929 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:49:39.915381 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:49:39.917461 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:39.917464 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:39.917466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:39.917982 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:39.918647 (+ 665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:39.918650 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:39.918652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:39.918739 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:39.919208 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:49:39.919210 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:49:39.919306 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 00:49:39.920343 (+ 1037us) client_negotiation.cc:240] Negotiation successful
0927 00:49:39.920513 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243}
Sep 27 00:49:39 dist-test-slave-86rs krb5kdc[13713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:49:39.921581 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.910263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48362 (local address 127.0.98.62:42023)
0927 00:49:39.910378 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:39.910382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:39.910394 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:49:39.910423 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:39.910425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:39.910465 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:39.910544 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:39.911405 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.911917 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.912789 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.913021 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.913132 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:39.916495 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:39.916500 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:39.916503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:39.916536 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:39.918977 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.919513 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.919516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.919518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.919617 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.919933 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.919936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.919938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.920117 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:39.920361 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:39.921289 (+ 928us) server_negotiation.cc:299] Negotiation successful
0927 00:49:39.921413 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:49:39.921523 13944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33419 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:49:39.922883 13794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c6a771b03054ac3bf6960ab8e51f72e: No bootstrap required, opened a new log
I20260927 00:49:39.925275 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.914873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33419 (local address 127.0.0.1:36278)
0927 00:49:39.915067 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:39.915083 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:39.915163 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:39.916612 (+ 1449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:39.916618 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:39.916893 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:49:39.916988 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:39.916994 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.917919 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.917922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:39.918764 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:39.918776 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.919389 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.919394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:39.919481 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:39.919897 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:49:39.919911 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:49:39.921469 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:49:39.923710 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:39.923713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:39.923715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:39.923965 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:39.924288 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:39.924291 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:39.924293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:39.924376 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:39.924786 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:49:39.924789 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:49:39.924880 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 00:49:39.924979 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 00:49:39.925109 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260927 00:49:39.925850 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.914932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36278 (local address 127.0.98.60:33419)
0927 00:49:39.916366 (+ 1434us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:39.916369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:39.916380 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:49:39.916408 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:39.916410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:39.916475 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:39.916778 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:39.917113 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.917782 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.918899 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.919261 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.919692 (+ 431us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:39.921636 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:39.921641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:39.921645 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:39.921678 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:39.923598 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.924065 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.924069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.924071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.924151 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.924470 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.924474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.924475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.924658 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:39.924744 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:39.925595 (+ 851us) server_negotiation.cc:299] Negotiation successful
0927 00:49:39.925704 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1354}
I20260927 00:49:39.927487 13794 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c6a771b03054ac3bf6960ab8e51f72e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a771b03054ac3bf6960ab8e51f72e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42023 } } peers { permanent_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39221 } } peers { permanent_uuid: "d83c3266fdd645a88a9c61c87d1e7387" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33419 } }
I20260927 00:49:39.927778 13794 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c6a771b03054ac3bf6960ab8e51f72e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:49:39.927907 13794 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c6a771b03054ac3bf6960ab8e51f72e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c6a771b03054ac3bf6960ab8e51f72e, State: Initialized, Role: FOLLOWER
Sep 27 00:49:39 dist-test-slave-86rs krb5kdc[13713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:49:39.928562 13794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c6a771b03054ac3bf6960ab8e51f72e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a771b03054ac3bf6960ab8e51f72e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42023 } } peers { permanent_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39221 } } peers { permanent_uuid: "d83c3266fdd645a88a9c61c87d1e7387" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33419 } }
I20260927 00:49:39.929263 13869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33: Bootstrap starting.
Sep 27 00:49:39 dist-test-slave-86rs krb5kdc[13713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:49:39.930233 13794 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c6a771b03054ac3bf6960ab8e51f72e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:49:39.932157 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.922691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33419 (local address 127.0.0.1:36290)
0927 00:49:39.922797 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:39.922810 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:39.922893 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:39.923322 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:39.923327 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:39.923645 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:49:39.923739 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:39.923746 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.924759 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.924764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:39.925773 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:39.925782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.926617 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.926620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:39.926688 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:39.927126 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:49:39.927139 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:49:39.928830 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:49:39.930735 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:39.930738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:39.930740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:39.930964 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:39.931227 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:39.931230 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:39.931231 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:39.931296 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:39.931686 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:49:39.931689 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:49:39.931764 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 00:49:39.931853 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 00:49:39.931984 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 00:49:39.932379 13869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33: Neither blocks nor log segments found. Creating new log.
I20260927 00:49:39.932822 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.922048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36290 (local address 127.0.98.60:33419)
0927 00:49:39.922504 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:39.922510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:39.922999 (+ 489us) server_negotiation.cc:407] Connection header received
0927 00:49:39.923102 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:39.923107 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:39.923152 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:39.923216 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:39.923847 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.924659 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.925888 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.926433 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.926524 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:39.928962 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:39.928966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:39.928968 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:39.928990 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:39.930609 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.931073 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.931075 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.931077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.931147 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.931372 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.931374 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.931376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.931566 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:39.931653 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:39.932564 (+ 911us) server_negotiation.cc:299] Negotiation successful
0927 00:49:39.932695 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 00:49:39.933521 13869 log.cc:824] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33: Log is configured to *not* fsync() on all Append() calls
I20260927 00:49:39.934912 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.923449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43122 (local address 127.0.98.61:39221)
0927 00:49:39.923564 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:39.923568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:39.923644 (+ 76us) server_negotiation.cc:407] Connection header received
0927 00:49:39.923808 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:39.923811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:39.923852 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:39.923900 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:39.924761 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.925467 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.926570 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.927218 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.927288 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:39.930764 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:39.930768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:39.930771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:39.930801 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:39.932581 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.933438 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.933442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.933443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.933532 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.933914 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.933917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.933919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.934112 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:39.934256 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:39.934637 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 00:49:39.934756 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":30}
I20260927 00:49:39.936187 13944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d83c3266fdd645a88a9c61c87d1e7387: Bootstrap starting.
W20260927 00:49:39.937505 13962 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c6a771b03054ac3bf6960ab8e51f72e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:49:39.937600 13962 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:49:39.937884 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c6a771b03054ac3bf6960ab8e51f72e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a771b03054ac3bf6960ab8e51f72e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42023 } } peers { permanent_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39221 } } peers { permanent_uuid: "d83c3266fdd645a88a9c61c87d1e7387" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33419 } } }
I20260927 00:49:39.938020 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c6a771b03054ac3bf6960ab8e51f72e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:49:39.938901 13944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d83c3266fdd645a88a9c61c87d1e7387: Neither blocks nor log segments found. Creating new log.
I20260927 00:49:39.939649 13944 log.cc:824] T 00000000000000000000000000000000 P d83c3266fdd645a88a9c61c87d1e7387: Log is configured to *not* fsync() on all Append() calls
I20260927 00:49:39.935783 13726 master_runner.cc:186] Error getting master registration for 127.0.98.61:39221: OK, instance_id { permanent_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" instance_seqno: 1790470179658594 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:49:39.939827 13726 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:49:39.941708 13944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d83c3266fdd645a88a9c61c87d1e7387: No bootstrap required, opened a new log
I20260927 00:49:39.942512 13869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33: No bootstrap required, opened a new log
I20260927 00:49:39.943791 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:49:39.946192 13944 raft_consensus.cc:348] T 00000000000000000000000000000000 P d83c3266fdd645a88a9c61c87d1e7387 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a771b03054ac3bf6960ab8e51f72e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42023 } } peers { permanent_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39221 } } peers { permanent_uuid: "d83c3266fdd645a88a9c61c87d1e7387" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33419 } }
I20260927 00:49:39.946475 13944 raft_consensus.cc:374] T 00000000000000000000000000000000 P d83c3266fdd645a88a9c61c87d1e7387 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:49:39.946534 13944 raft_consensus.cc:729] T 00000000000000000000000000000000 P d83c3266fdd645a88a9c61c87d1e7387 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d83c3266fdd645a88a9c61c87d1e7387, State: Initialized, Role: FOLLOWER
I20260927 00:49:39.947122 13944 consensus_queue.cc:260] T 00000000000000000000000000000000 P d83c3266fdd645a88a9c61c87d1e7387 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a771b03054ac3bf6960ab8e51f72e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42023 } } peers { permanent_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39221 } } peers { permanent_uuid: "d83c3266fdd645a88a9c61c87d1e7387" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33419 } }
I20260927 00:49:39.948686 13944 sys_catalog.cc:564] T 00000000000000000000000000000000 P d83c3266fdd645a88a9c61c87d1e7387 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:49:39.949731 13869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a771b03054ac3bf6960ab8e51f72e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42023 } } peers { permanent_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39221 } } peers { permanent_uuid: "d83c3266fdd645a88a9c61c87d1e7387" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33419 } }
I20260927 00:49:39.949999 13869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:49:39.950065 13869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fb5f67ae39e4c0a852804f4aacb2d33, State: Initialized, Role: FOLLOWER
I20260927 00:49:39.950698 13869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a771b03054ac3bf6960ab8e51f72e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42023 } } peers { permanent_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39221 } } peers { permanent_uuid: "d83c3266fdd645a88a9c61c87d1e7387" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33419 } }
I20260927 00:49:39.951381 13970 sys_catalog.cc:455] T 00000000000000000000000000000000 P d83c3266fdd645a88a9c61c87d1e7387 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a771b03054ac3bf6960ab8e51f72e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42023 } } peers { permanent_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39221 } } peers { permanent_uuid: "d83c3266fdd645a88a9c61c87d1e7387" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33419 } } }
I20260927 00:49:39.951490 13970 sys_catalog.cc:458] T 00000000000000000000000000000000 P d83c3266fdd645a88a9c61c87d1e7387 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:49:39.952749 13869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:49:39.959543 13875 master_runner.cc:263] No leader master found from master d83c3266fdd645a88a9c61c87d1e7387
I20260927 00:49:39.959621 13875 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 00:49:39.961149 13982 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d83c3266fdd645a88a9c61c87d1e7387: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:49:39.961323 13982 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:49:39.961447 13801 master_runner.cc:263] No leader master found from master 4fb5f67ae39e4c0a852804f4aacb2d33
I20260927 00:49:39.961529 13801 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:49:39.961790 13983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a771b03054ac3bf6960ab8e51f72e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42023 } } peers { permanent_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39221 } } peers { permanent_uuid: "d83c3266fdd645a88a9c61c87d1e7387" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33419 } } }
I20260927 00:49:39.961997 13983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:49:39.963855 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.948263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48378 (local address 127.0.98.62:42023)
0927 00:49:39.948400 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:39.948405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:39.951820 (+ 3415us) server_negotiation.cc:407] Connection header received
0927 00:49:39.952030 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:39.952034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:39.952081 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:39.952136 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:39.953215 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.953955 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.957134 (+ 3179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.957453 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.957533 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:39.959538 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:39.959542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:39.959545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:39.959615 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:39.961722 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.962224 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.962228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.962233 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.962331 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.962843 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.962847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.962849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.963087 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:39.963212 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:39.963575 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 00:49:39.963705 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 00:49:39.965883 13994 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:49:39.965981 13994 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:49:39.974854 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.956263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43132 (local address 127.0.98.61:39221)
0927 00:49:39.956503 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:39.956507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:39.965144 (+ 8637us) server_negotiation.cc:407] Connection header received
0927 00:49:39.965334 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:39.965337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:39.965385 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:39.965436 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:39.966459 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.967295 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.968516 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.968860 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.968956 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:39.970911 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:39.970917 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:39.970922 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:39.970961 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:39.972754 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.973292 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.973296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.973297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.973390 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.973737 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.973741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.973744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.973988 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:39.974127 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:39.974509 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 00:49:39.974640 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260927 00:49:39.976173 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:39.955635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36292 (local address 127.0.98.60:33419)
0927 00:49:39.955749 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:39.955754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:39.966295 (+ 10541us) server_negotiation.cc:407] Connection header received
0927 00:49:39.966563 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:39.966566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:39.966618 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:39.966669 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:39.967542 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.968312 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.969418 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:39.970114 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:39.970207 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:39.972596 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:39.972602 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:39.972608 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:39.972641 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:39.974330 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.974803 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.974806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.974808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.974891 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:39.975213 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:39.975216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:39.975218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:39.975427 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:39.975525 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:39.975857 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 00:49:39.975979 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:49:40.120033 13983 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:49:40.120388 13983 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a771b03054ac3bf6960ab8e51f72e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42023 } } peers { permanent_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39221 } } peers { permanent_uuid: "d83c3266fdd645a88a9c61c87d1e7387" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33419 } }
I20260927 00:49:40.121636 13983 leader_election.cc:288] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c6a771b03054ac3bf6960ab8e51f72e (127.0.98.62:42023), d83c3266fdd645a88a9c61c87d1e7387 (127.0.98.60:33419)
I20260927 00:49:40.122623 13768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c6a771b03054ac3bf6960ab8e51f72e" is_pre_election: true
I20260927 00:49:40.123001 13768 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9c6a771b03054ac3bf6960ab8e51f72e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fb5f67ae39e4c0a852804f4aacb2d33 in term 0.
I20260927 00:49:40.123734 13817 leader_election.cc:302] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fb5f67ae39e4c0a852804f4aacb2d33, 9c6a771b03054ac3bf6960ab8e51f72e; no voters:
I20260927 00:49:40.124145 13983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:49:40.124258 13983 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:49:40.124303 13983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:49:40.125608 13983 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a771b03054ac3bf6960ab8e51f72e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42023 } } peers { permanent_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39221 } } peers { permanent_uuid: "d83c3266fdd645a88a9c61c87d1e7387" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33419 } }
I20260927 00:49:40.126266 13983 leader_election.cc:288] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33 [CANDIDATE]: Term 1 election: Requested vote from peers 9c6a771b03054ac3bf6960ab8e51f72e (127.0.98.62:42023), d83c3266fdd645a88a9c61c87d1e7387 (127.0.98.60:33419)
I20260927 00:49:40.126801 13768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c6a771b03054ac3bf6960ab8e51f72e"
I20260927 00:49:40.126942 13768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c6a771b03054ac3bf6960ab8e51f72e [term 0 FOLLOWER]: Advancing to term 1
W20260927 00:49:40.127843 13897 tablet.cc:2369] T 00000000000000000000000000000000 P d83c3266fdd645a88a9c61c87d1e7387: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:49:40.128683 13768 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9c6a771b03054ac3bf6960ab8e51f72e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fb5f67ae39e4c0a852804f4aacb2d33 in term 1.
I20260927 00:49:40.129287 13817 leader_election.cc:302] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fb5f67ae39e4c0a852804f4aacb2d33, 9c6a771b03054ac3bf6960ab8e51f72e; no voters:
I20260927 00:49:40.129873 13918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d83c3266fdd645a88a9c61c87d1e7387" is_pre_election: true
I20260927 00:49:40.130272 13918 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d83c3266fdd645a88a9c61c87d1e7387 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fb5f67ae39e4c0a852804f4aacb2d33 in term 0.
I20260927 00:49:40.130894 13983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:49:40.131165 13983 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33 [term 1 LEADER]: Becoming Leader. State: Replica: 4fb5f67ae39e4c0a852804f4aacb2d33, State: Running, Role: LEADER
I20260927 00:49:40.131650 13983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a771b03054ac3bf6960ab8e51f72e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42023 } } peers { permanent_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39221 } } peers { permanent_uuid: "d83c3266fdd645a88a9c61c87d1e7387" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33419 } }
I20260927 00:49:40.132369 13918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d83c3266fdd645a88a9c61c87d1e7387"
I20260927 00:49:40.132503 13918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d83c3266fdd645a88a9c61c87d1e7387 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:49:40.133898 13918 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d83c3266fdd645a88a9c61c87d1e7387 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fb5f67ae39e4c0a852804f4aacb2d33 in term 1.
I20260927 00:49:40.136338 13983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fb5f67ae39e4c0a852804f4aacb2d33. Latest consensus state: current_term: 1 leader_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a771b03054ac3bf6960ab8e51f72e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42023 } } peers { permanent_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39221 } } peers { permanent_uuid: "d83c3266fdd645a88a9c61c87d1e7387" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33419 } } }
I20260927 00:49:40.136569 13983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33 [sys.catalog]: This master's current role is: LEADER
I20260927 00:49:40.139948 14001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:49:40.141007 14001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:49:40.147012 13768 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9c6a771b03054ac3bf6960ab8e51f72e [term 1 FOLLOWER]: Refusing update from remote peer 4fb5f67ae39e4c0a852804f4aacb2d33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:49:40.147012 13918 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d83c3266fdd645a88a9c61c87d1e7387 [term 1 FOLLOWER]: Refusing update from remote peer 4fb5f67ae39e4c0a852804f4aacb2d33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:49:40.147907 13999 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d83c3266fdd645a88a9c61c87d1e7387" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:49:40.148294 13983 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c6a771b03054ac3bf6960ab8e51f72e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:49:40.152537 13970 sys_catalog.cc:455] T 00000000000000000000000000000000 P d83c3266fdd645a88a9c61c87d1e7387 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fb5f67ae39e4c0a852804f4aacb2d33. Latest consensus state: current_term: 1 leader_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a771b03054ac3bf6960ab8e51f72e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42023 } } peers { permanent_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39221 } } peers { permanent_uuid: "d83c3266fdd645a88a9c61c87d1e7387" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33419 } } }
I20260927 00:49:40.152537 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c6a771b03054ac3bf6960ab8e51f72e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fb5f67ae39e4c0a852804f4aacb2d33. Latest consensus state: current_term: 1 leader_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a771b03054ac3bf6960ab8e51f72e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42023 } } peers { permanent_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39221 } } peers { permanent_uuid: "d83c3266fdd645a88a9c61c87d1e7387" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33419 } } }
I20260927 00:49:40.152686 13970 sys_catalog.cc:458] T 00000000000000000000000000000000 P d83c3266fdd645a88a9c61c87d1e7387 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:49:40.152686 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c6a771b03054ac3bf6960ab8e51f72e [sys.catalog]: This master's current role is: FOLLOWER
W20260927 00:49:40.154930 13822 tablet.cc:2369] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:49:40.157461 13983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a771b03054ac3bf6960ab8e51f72e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42023 } } peers { permanent_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39221 } } peers { permanent_uuid: "d83c3266fdd645a88a9c61c87d1e7387" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33419 } } }
I20260927 00:49:40.157594 13983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33 [sys.catalog]: This master's current role is: LEADER
I20260927 00:49:40.158005 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c6a771b03054ac3bf6960ab8e51f72e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a771b03054ac3bf6960ab8e51f72e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42023 } } peers { permanent_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39221 } } peers { permanent_uuid: "d83c3266fdd645a88a9c61c87d1e7387" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33419 } } }
I20260927 00:49:40.158126 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c6a771b03054ac3bf6960ab8e51f72e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:49:40.158789 14004 mvcc.cc:205] Tried to move back new op lower bound from 7333765857874149376 to 7333765857833766912. Current Snapshot: MvccSnapshot[applied={T|T < 7333765857874149376}]
I20260927 00:49:40.160017 13999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a771b03054ac3bf6960ab8e51f72e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42023 } } peers { permanent_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39221 } } peers { permanent_uuid: "d83c3266fdd645a88a9c61c87d1e7387" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33419 } } }
I20260927 00:49:40.160143 13999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33 [sys.catalog]: This master's current role is: LEADER
I20260927 00:49:40.161360 14001 catalog_manager.cc:1347] Generated new cluster ID: e9a3179454f04f8187df20039184955c
I20260927 00:49:40.161444 14001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:49:40.161650 13970 sys_catalog.cc:455] T 00000000000000000000000000000000 P d83c3266fdd645a88a9c61c87d1e7387 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6a771b03054ac3bf6960ab8e51f72e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42023 } } peers { permanent_uuid: "4fb5f67ae39e4c0a852804f4aacb2d33" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39221 } } peers { permanent_uuid: "d83c3266fdd645a88a9c61c87d1e7387" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33419 } } }
I20260927 00:49:40.161782 13970 sys_catalog.cc:458] T 00000000000000000000000000000000 P d83c3266fdd645a88a9c61c87d1e7387 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:49:40.177906 14001 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:49:40.178944 14001 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:49:40.186456 14001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fb5f67ae39e4c0a852804f4aacb2d33: Generated new TSK 0
I20260927 00:49:40.187597 14001 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:49:40.939020 13962 catalog_manager.cc:1259] Loaded cluster ID: e9a3179454f04f8187df20039184955c
I20260927 00:49:40.939112 13962 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9c6a771b03054ac3bf6960ab8e51f72e: loading cluster ID for follower catalog manager: success
I20260927 00:49:40.940779 13962 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9c6a771b03054ac3bf6960ab8e51f72e: acquiring CA information for follower catalog manager: success
I20260927 00:49:40.941937 13962 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9c6a771b03054ac3bf6960ab8e51f72e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:49:40.962932 13982 catalog_manager.cc:1259] Loaded cluster ID: e9a3179454f04f8187df20039184955c
I20260927 00:49:40.963022 13982 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d83c3266fdd645a88a9c61c87d1e7387: loading cluster ID for follower catalog manager: success
I20260927 00:49:40.964848 13982 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d83c3266fdd645a88a9c61c87d1e7387: acquiring CA information for follower catalog manager: success
I20260927 00:49:40.966043 13982 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d83c3266fdd645a88a9c61c87d1e7387: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:49:41.247118 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 13726
I20260927 00:49:41.254079 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 13801
I20260927 00:49:41.260565 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 13875
2026-09-27T00:49:41Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3213 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:49:41 dist-test-slave-86rs krb5kdc[14016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:49:41 dist-test-slave-86rs krb5kdc[14016](info): set up 2 sockets
Sep 27 00:49:41 dist-test-slave-86rs krb5kdc[14016](info): commencing operation
krb5kdc: starting...
W20260927 00:49:42.296542 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:49:42 dist-test-slave-86rs krb5kdc[14016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:49:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:49:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:49:42.424155 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44219
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:49:42.540019 14029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:49:42.540414 14029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:49:42.540498 14029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:49:42.544631 14029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:49:42.544701 14029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:49:42.544723 14029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:49:42.544739 14029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:49:42.544754 14029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:49:42.550539 14029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:49:42.551746 14029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:49:42.552896 14029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:49:42.562363 14034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:42.562378 14035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:42.562363 14037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:42.562757 14029 server_base.cc:1034] running on GCE node
I20260927 00:49:42.563334 14029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:49:42.564667 14029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:49:42.565830 14029 hybrid_clock.cc:648] HybridClock initialized: now 1790470182565813 us; error 34 us; skew 500 ppm
Sep 27 00:49:42 dist-test-slave-86rs krb5kdc[14016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:49:42.568750 14029 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:49:42.569651 14029 webserver.cc:460] Webserver started at http://127.0.98.62:34617/ using document root <none> and password file <none>
I20260927 00:49:42.570221 14029 fs_manager.cc:359] Metadata directory not provided
I20260927 00:49:42.570283 14029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:49:42.570453 14029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:49:42.572685 14029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "5e08eaf035e742a1bdbd258b5c98eef9"
format_stamp: "Formatted at 2026-09-27 00:49:42 on dist-test-slave-86rs"
server_key: "13c5618afb5ec036475b73131fe1506c"
server_key_iv: "f832fc77ab25c98de29d21038728b88f"
server_key_version: "encryptionkey@0"
I20260927 00:49:42.573223 14029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "5e08eaf035e742a1bdbd258b5c98eef9"
format_stamp: "Formatted at 2026-09-27 00:49:42 on dist-test-slave-86rs"
server_key: "13c5618afb5ec036475b73131fe1506c"
server_key_iv: "f832fc77ab25c98de29d21038728b88f"
server_key_version: "encryptionkey@0"
I20260927 00:49:42.577345 14029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:49:42.579986 14044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:42.581409 14029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:49:42.581526 14029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "5e08eaf035e742a1bdbd258b5c98eef9"
format_stamp: "Formatted at 2026-09-27 00:49:42 on dist-test-slave-86rs"
server_key: "13c5618afb5ec036475b73131fe1506c"
server_key_iv: "f832fc77ab25c98de29d21038728b88f"
server_key_version: "encryptionkey@0"
I20260927 00:49:42.581651 14029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:49:42.623036 14029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:49:42.626006 14029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:49:42.626166 14029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:49:42.633852 14029 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44219
I20260927 00:49:42.633875 14096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44219 every 8 connection(s)
I20260927 00:49:42.634918 14029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:49:42.637936 14097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:49:42.641660 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 14029
I20260927 00:49:42.641769 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:49:42.642079 392 external_mini_cluster.cc:1428] Setting key 39ef4ba0d174ea1c6d71593935cb7a46
I20260927 00:49:42.644294 14097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e08eaf035e742a1bdbd258b5c98eef9: Bootstrap starting.
I20260927 00:49:42.647135 14097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e08eaf035e742a1bdbd258b5c98eef9: Neither blocks nor log segments found. Creating new log.
I20260927 00:49:42.648056 14097 log.cc:824] T 00000000000000000000000000000000 P 5e08eaf035e742a1bdbd258b5c98eef9: Log is configured to *not* fsync() on all Append() calls
I20260927 00:49:42.650547 14097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e08eaf035e742a1bdbd258b5c98eef9: No bootstrap required, opened a new log
Sep 27 00:49:42 dist-test-slave-86rs krb5kdc[14016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:49:42.654557 14097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e08eaf035e742a1bdbd258b5c98eef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e08eaf035e742a1bdbd258b5c98eef9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44219 } }
I20260927 00:49:42.654865 14097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e08eaf035e742a1bdbd258b5c98eef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:49:42.654917 14097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e08eaf035e742a1bdbd258b5c98eef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e08eaf035e742a1bdbd258b5c98eef9, State: Initialized, Role: FOLLOWER
I20260927 00:49:42.655460 14097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e08eaf035e742a1bdbd258b5c98eef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e08eaf035e742a1bdbd258b5c98eef9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44219 } }
I20260927 00:49:42.655596 14097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e08eaf035e742a1bdbd258b5c98eef9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:49:42.655678 14097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e08eaf035e742a1bdbd258b5c98eef9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:49:42.655798 14097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e08eaf035e742a1bdbd258b5c98eef9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:49:42.657045 14097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e08eaf035e742a1bdbd258b5c98eef9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e08eaf035e742a1bdbd258b5c98eef9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44219 } }
I20260927 00:49:42.657512 14097 leader_election.cc:302] T 00000000000000000000000000000000 P 5e08eaf035e742a1bdbd258b5c98eef9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e08eaf035e742a1bdbd258b5c98eef9; no voters:
I20260927 00:49:42.657899 14097 leader_election.cc:288] T 00000000000000000000000000000000 P 5e08eaf035e742a1bdbd258b5c98eef9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:49:42.658092 14102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e08eaf035e742a1bdbd258b5c98eef9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:49:42.658382 14102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e08eaf035e742a1bdbd258b5c98eef9 [term 1 LEADER]: Becoming Leader. State: Replica: 5e08eaf035e742a1bdbd258b5c98eef9, State: Running, Role: LEADER
I20260927 00:49:42.658761 14102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e08eaf035e742a1bdbd258b5c98eef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e08eaf035e742a1bdbd258b5c98eef9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44219 } }
I20260927 00:49:42.659190 14100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:42.643516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58872 (local address 127.0.98.62:44219)
0927 00:49:42.643974 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:42.643994 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:42.644038 (+ 44us) server_negotiation.cc:407] Connection header received
0927 00:49:42.644894 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:42.644918 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:42.645348 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:42.645743 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:42.646439 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:42.647978 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:42.649179 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:42.649900 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:42.650067 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:42.652633 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:42.652640 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:42.652658 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:42.652692 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:42.655282 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:42.655953 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:42.655962 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:42.655968 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:42.656111 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:42.656494 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:42.656498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:42.656499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:42.657100 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:42.657299 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:42.657684 (+ 385us) server_negotiation.cc:299] Negotiation successful
0927 00:49:42.658137 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":121,"threads_started":1}
I20260927 00:49:42.659492 14097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e08eaf035e742a1bdbd258b5c98eef9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:49:42.660562 14104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e08eaf035e742a1bdbd258b5c98eef9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e08eaf035e742a1bdbd258b5c98eef9. Latest consensus state: current_term: 1 leader_uuid: "5e08eaf035e742a1bdbd258b5c98eef9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e08eaf035e742a1bdbd258b5c98eef9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44219 } } }
I20260927 00:49:42.660679 14104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e08eaf035e742a1bdbd258b5c98eef9 [sys.catalog]: This master's current role is: LEADER
I20260927 00:49:42.660976 14103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e08eaf035e742a1bdbd258b5c98eef9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e08eaf035e742a1bdbd258b5c98eef9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e08eaf035e742a1bdbd258b5c98eef9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44219 } } }
I20260927 00:49:42.661082 14103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e08eaf035e742a1bdbd258b5c98eef9 [sys.catalog]: This master's current role is: LEADER
I20260927 00:49:42.661111 14107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:49:42.662505 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:49:42.664565 14107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:49:42.671128 14107 catalog_manager.cc:1347] Generated new cluster ID: 15ff365f8088463ab603c9c691c9a9f2
I20260927 00:49:42.671252 14107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:49:42.683804 14107 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:49:42.684975 14107 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:49:42.690730 14107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e08eaf035e742a1bdbd258b5c98eef9: Generated new TSK 0
I20260927 00:49:42.691570 14107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:49:42.719457 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 14029
2026-09-27T00:49:42Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1456 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:49:42 dist-test-slave-86rs krb5kdc[14124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:49:42 dist-test-slave-86rs krb5kdc[14124](info): set up 2 sockets
Sep 27 00:49:42 dist-test-slave-86rs krb5kdc[14124](info): commencing operation
krb5kdc: starting...
W20260927 00:49:43.756711 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:49:43 dist-test-slave-86rs krb5kdc[14124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:49:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:49:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:49:43.884132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34607
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:49:44.003237 14137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:49:44.003551 14137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:49:44.003597 14137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:49:44.007870 14137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:49:44.007964 14137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:49:44.007985 14137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:49:44.008002 14137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:49:44.008018 14137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:49:44.014144 14137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:49:44.015344 14137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:49:44.016438 14137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:49:44.025583 14145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:44.025569 14142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:44.025569 14143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:44.025784 14137 server_base.cc:1034] running on GCE node
I20260927 00:49:44.026628 14137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:49:44.027864 14137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:49:44.029058 14137 hybrid_clock.cc:648] HybridClock initialized: now 1790470184029014 us; error 62 us; skew 500 ppm
Sep 27 00:49:44 dist-test-slave-86rs krb5kdc[14124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:49:44.036067 14137 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:49:44.037567 14137 webserver.cc:460] Webserver started at http://127.0.98.62:45967/ using document root <none> and password file <none>
I20260927 00:49:44.038337 14137 fs_manager.cc:359] Metadata directory not provided
I20260927 00:49:44.038407 14137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:49:44.038655 14137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:49:44.041059 14137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "98ab283e0f91492083a657e1243d264b"
format_stamp: "Formatted at 2026-09-27 00:49:44 on dist-test-slave-86rs"
server_key: "dc0812c066da0be98532959f6ea74f55"
server_key_iv: "a9c0c0cce32783683fcd272a80019b63"
server_key_version: "encryptionkey@0"
I20260927 00:49:44.041661 14137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "98ab283e0f91492083a657e1243d264b"
format_stamp: "Formatted at 2026-09-27 00:49:44 on dist-test-slave-86rs"
server_key: "dc0812c066da0be98532959f6ea74f55"
server_key_iv: "a9c0c0cce32783683fcd272a80019b63"
server_key_version: "encryptionkey@0"
I20260927 00:49:44.045773 14137 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:49:44.048532 14152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:44.049882 14137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:49:44.050118 14137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "98ab283e0f91492083a657e1243d264b"
format_stamp: "Formatted at 2026-09-27 00:49:44 on dist-test-slave-86rs"
server_key: "dc0812c066da0be98532959f6ea74f55"
server_key_iv: "a9c0c0cce32783683fcd272a80019b63"
server_key_version: "encryptionkey@0"
I20260927 00:49:44.050246 14137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:49:44.066075 14137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:49:44.069767 14137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:49:44.069978 14137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:49:44.077517 14137 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34607
I20260927 00:49:44.077813 14204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34607 every 8 connection(s)
I20260927 00:49:44.078630 14137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:49:44.081497 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 14137
I20260927 00:49:44.081676 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:49:44.081902 14205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:49:44.082026 392 external_mini_cluster.cc:1428] Setting key f62238ea4cf021c3af18bfb5448d657f
I20260927 00:49:44.089928 14205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b: Bootstrap starting.
Sep 27 00:49:44 dist-test-slave-86rs krb5kdc[14124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:49:44.092681 14205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b: Neither blocks nor log segments found. Creating new log.
I20260927 00:49:44.093842 14205 log.cc:824] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b: Log is configured to *not* fsync() on all Append() calls
I20260927 00:49:44.096639 14205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b: No bootstrap required, opened a new log
I20260927 00:49:44.098765 14208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:44.083534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47312 (local address 127.0.98.62:34607)
0927 00:49:44.084028 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:44.084050 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:44.084176 (+ 126us) server_negotiation.cc:407] Connection header received
0927 00:49:44.085029 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:44.085058 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:44.085546 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:44.085975 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:44.086681 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:44.088212 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:44.089191 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:44.089900 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:44.090088 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:44.093119 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:44.093126 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:44.093145 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:44.093185 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:44.095273 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:44.095883 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:44.095889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:44.095894 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:44.096028 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:44.096432 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:44.096435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:44.096436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:44.096984 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:44.097159 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:44.097564 (+ 405us) server_negotiation.cc:299] Negotiation successful
0927 00:49:44.097968 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":129,"threads_started":1}
I20260927 00:49:44.100965 14205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ab283e0f91492083a657e1243d264b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34607 } }
I20260927 00:49:44.101262 14205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:49:44.101322 14205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98ab283e0f91492083a657e1243d264b, State: Initialized, Role: FOLLOWER
I20260927 00:49:44.101816 14205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ab283e0f91492083a657e1243d264b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34607 } }
I20260927 00:49:44.101979 14205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:49:44.102069 14205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:49:44.102195 14205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:49:44.103216 14205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ab283e0f91492083a657e1243d264b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34607 } }
I20260927 00:49:44.103585 14205 leader_election.cc:302] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98ab283e0f91492083a657e1243d264b; no voters:
I20260927 00:49:44.103941 14205 leader_election.cc:288] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:49:44.104091 14210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:49:44.104388 14210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [term 1 LEADER]: Becoming Leader. State: Replica: 98ab283e0f91492083a657e1243d264b, State: Running, Role: LEADER
I20260927 00:49:44.104734 14210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ab283e0f91492083a657e1243d264b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34607 } }
I20260927 00:49:44.105415 14205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:49:44.106652 14212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98ab283e0f91492083a657e1243d264b. Latest consensus state: current_term: 1 leader_uuid: "98ab283e0f91492083a657e1243d264b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ab283e0f91492083a657e1243d264b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34607 } } }
I20260927 00:49:44.106801 14212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [sys.catalog]: This master's current role is: LEADER
I20260927 00:49:44.107124 14211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98ab283e0f91492083a657e1243d264b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ab283e0f91492083a657e1243d264b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34607 } } }
I20260927 00:49:44.107218 14211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [sys.catalog]: This master's current role is: LEADER
I20260927 00:49:44.107326 14219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:49:44.110705 14219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:49:44.116901 14219 catalog_manager.cc:1347] Generated new cluster ID: ca38794ed4a543a188b45b3426c8cee2
I20260927 00:49:44.117009 14219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:49:44.129739 14219 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:49:44.130789 14219 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:49:44.137655 14219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b: Generated new TSK 0
I20260927 00:49:44.138527 14219 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:49:44.171617 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34607
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:49:44.290467 14231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:49:44.290776 14231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:49:44.290823 14231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:49:44.295089 14231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:49:44.295177 14231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:49:44.295241 14231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:49:44.300886 14231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:49:44.302039 14231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:49:44.303089 14231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:49:44.313669 14236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:44.313884 14231 server_base.cc:1034] running on GCE node
W20260927 00:49:44.313669 14239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:44.313745 14237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:44.314476 14231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:49:44.315135 14231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:49:44.316303 14231 hybrid_clock.cc:648] HybridClock initialized: now 1790470184316260 us; error 49 us; skew 500 ppm
Sep 27 00:49:44 dist-test-slave-86rs krb5kdc[14124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:49:44.319141 14231 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:49:44.320091 14231 webserver.cc:460] Webserver started at http://127.0.98.1:35185/ using document root <none> and password file <none>
I20260927 00:49:44.320712 14231 fs_manager.cc:359] Metadata directory not provided
I20260927 00:49:44.320775 14231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:49:44.321008 14231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:49:44.323186 14231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/data/instance:
uuid: "e5bfcca01792442087b4108656ab072c"
format_stamp: "Formatted at 2026-09-27 00:49:44 on dist-test-slave-86rs"
server_key: "67d708c3b58b8ffa4af2715c4812134f"
server_key_iv: "5eb5d584d11a602dec080be37068f7a2"
server_key_version: "encryptionkey@0"
I20260927 00:49:44.323791 14231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e5bfcca01792442087b4108656ab072c"
format_stamp: "Formatted at 2026-09-27 00:49:44 on dist-test-slave-86rs"
server_key: "67d708c3b58b8ffa4af2715c4812134f"
server_key_iv: "5eb5d584d11a602dec080be37068f7a2"
server_key_version: "encryptionkey@0"
I20260927 00:49:44.327781 14231 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:49:44.330549 14246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:44.331959 14231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:49:44.332125 14231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "e5bfcca01792442087b4108656ab072c"
format_stamp: "Formatted at 2026-09-27 00:49:44 on dist-test-slave-86rs"
server_key: "67d708c3b58b8ffa4af2715c4812134f"
server_key_iv: "5eb5d584d11a602dec080be37068f7a2"
server_key_version: "encryptionkey@0"
I20260927 00:49:44.332280 14231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:49:44.344683 14231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:49:44.347714 14231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:49:44.347923 14231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:49:44.348639 14231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:49:44.349491 14231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:49:44.349572 14231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:44.349649 14231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:49:44.349704 14231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:44.360524 14231 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43081
I20260927 00:49:44.360536 14359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43081 every 8 connection(s)
I20260927 00:49:44.361523 14231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:49:44.368848 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 14231
I20260927 00:49:44.368996 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:49:44.369298 392 external_mini_cluster.cc:1428] Setting key 4dfd22e99fa1a5d060d85b7662383965
Sep 27 00:49:44 dist-test-slave-86rs krb5kdc[14124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:49:44.377130 14208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:44.363512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39425 (local address 127.0.98.62:34607)
0927 00:49:44.363656 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:44.363660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:44.364582 (+ 922us) server_negotiation.cc:407] Connection header received
0927 00:49:44.365915 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:44.365919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:44.365982 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:44.366092 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:44.367354 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:44.367928 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:44.369664 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:44.369915 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:44.370007 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:44.372782 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:44.372788 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:44.372794 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:44.372832 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:44.374610 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:44.375244 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:44.375247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:44.375248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:44.375337 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:44.375735 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:44.375739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:44.375742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:44.375959 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:44.376096 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:44.376782 (+ 686us) server_negotiation.cc:299] Negotiation successful
0927 00:49:44.376933 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:49:44.378024 14362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:44.363873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.1:39425)
0927 00:49:44.364404 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:44.364450 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:44.365690 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:44.366282 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:44.366289 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:44.366677 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:49:44.367187 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:44.367201 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:44.368069 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:44.368073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:44.369501 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:44.369515 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:44.370081 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:44.370085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:44.370270 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:44.370927 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:49:44.370944 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:49:44.372576 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:49:44.374759 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:44.374765 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:44.374781 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:44.375094 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:44.375447 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:44.375451 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:44.375453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:44.375620 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:44.376091 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:49:44.376098 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:49:44.376510 (+ 412us) client_negotiation.cc:769] Sending connection context
0927 00:49:44.376720 (+ 210us) client_negotiation.cc:240] Negotiation successful
0927 00:49:44.377103 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":126,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:49:44.379515 14360 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34607
I20260927 00:49:44.379765 14360 heartbeater.cc:464] Registering TS with master...
I20260927 00:49:44.380395 14360 heartbeater.cc:511] Master 127.0.98.62:34607 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:49:44.382228 14169 ts_manager.cc:194] Registered new tserver with Master: e5bfcca01792442087b4108656ab072c (127.0.98.1:43081)
I20260927 00:49:44.383452 14169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:49:44.398767 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34607
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:49:44.512964 14365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:49:44.513268 14365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:49:44.513315 14365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:49:44.517673 14365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:49:44.517750 14365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:49:44.517849 14365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:49:44.527158 14365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:49:44.528671 14365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:49:44.530010 14365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:49:44.541961 14371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:44.541961 14373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:44.542078 14370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:44.542295 14365 server_base.cc:1034] running on GCE node
I20260927 00:49:44.542860 14365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:49:44.543745 14365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:49:44.544940 14365 hybrid_clock.cc:648] HybridClock initialized: now 1790470184544902 us; error 52 us; skew 500 ppm
Sep 27 00:49:44 dist-test-slave-86rs krb5kdc[14124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:49:44.548794 14365 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:49:44.550050 14365 webserver.cc:460] Webserver started at http://127.0.98.2:33755/ using document root <none> and password file <none>
I20260927 00:49:44.550941 14365 fs_manager.cc:359] Metadata directory not provided
I20260927 00:49:44.551049 14365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:49:44.551363 14365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:49:44.553993 14365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/data/instance:
uuid: "06b92c2cf7e64899a2bfce641404ad87"
format_stamp: "Formatted at 2026-09-27 00:49:44 on dist-test-slave-86rs"
server_key: "bf15d5a1b31ea7bddf7f3df1fdf6d7d2"
server_key_iv: "c7b397315648031aa244b245ca9eafed"
server_key_version: "encryptionkey@0"
I20260927 00:49:44.554647 14365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/wal/instance:
uuid: "06b92c2cf7e64899a2bfce641404ad87"
format_stamp: "Formatted at 2026-09-27 00:49:44 on dist-test-slave-86rs"
server_key: "bf15d5a1b31ea7bddf7f3df1fdf6d7d2"
server_key_iv: "c7b397315648031aa244b245ca9eafed"
server_key_version: "encryptionkey@0"
I20260927 00:49:44.558863 14365 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:49:44.562503 14380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:44.564317 14365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 00:49:44.564469 14365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "06b92c2cf7e64899a2bfce641404ad87"
format_stamp: "Formatted at 2026-09-27 00:49:44 on dist-test-slave-86rs"
server_key: "bf15d5a1b31ea7bddf7f3df1fdf6d7d2"
server_key_iv: "c7b397315648031aa244b245ca9eafed"
server_key_version: "encryptionkey@0"
I20260927 00:49:44.564615 14365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:49:44.578438 14365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:49:44.582149 14365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:49:44.582391 14365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:49:44.583055 14365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:49:44.583935 14365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:49:44.584025 14365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:44.584120 14365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:49:44.584173 14365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:44.593147 14365 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35409
I20260927 00:49:44.593168 14493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35409 every 8 connection(s)
I20260927 00:49:44.594143 14365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:49:44.596020 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 14365
I20260927 00:49:44.596163 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:49:44.596570 392 external_mini_cluster.cc:1428] Setting key 953fff8b99348d97f55517dbd7dcfdf8
Sep 27 00:49:44 dist-test-slave-86rs krb5kdc[14124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:49:44.610989 14208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:44.596147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49199 (local address 127.0.98.62:34607)
0927 00:49:44.596297 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:44.596302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:44.597539 (+ 1237us) server_negotiation.cc:407] Connection header received
0927 00:49:44.599114 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:44.599121 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:44.599186 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:44.599281 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:44.600649 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:44.601197 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:44.603058 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:44.603297 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:44.603386 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:44.606813 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:44.606820 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:44.606827 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:44.606863 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:44.608501 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:44.609212 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:44.609215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:44.609216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:44.609299 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:44.609621 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:44.609623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:44.609624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:44.609897 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:44.609978 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:44.610648 (+ 670us) server_negotiation.cc:299] Negotiation successful
0927 00:49:44.610799 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 00:49:44.613260 14496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:44.596770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.2:49199)
0927 00:49:44.597359 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:44.597406 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:44.598881 (+ 1475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:44.599454 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:44.599462 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:44.599907 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:49:44.600494 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:44.600509 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:44.601334 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:44.601341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:44.602892 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:44.602906 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:44.603476 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:44.603481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:44.603735 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:44.604523 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:49:44.604560 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:49:44.606587 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:49:44.608777 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:44.608783 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:44.608819 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:44.609103 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:44.609381 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:44.609383 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:44.609384 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:44.609529 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:44.609993 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:49:44.609998 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:49:44.610361 (+ 363us) client_negotiation.cc:769] Sending connection context
0927 00:49:44.610584 (+ 223us) client_negotiation.cc:240] Negotiation successful
0927 00:49:44.612379 (+ 1795us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":165,"threads_started":1}
I20260927 00:49:44.614904 14494 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34607
I20260927 00:49:44.615204 14494 heartbeater.cc:464] Registering TS with master...
I20260927 00:49:44.615772 14494 heartbeater.cc:511] Master 127.0.98.62:34607 requested a full tablet report, sending...
I20260927 00:49:44.617059 14169 ts_manager.cc:194] Registered new tserver with Master: 06b92c2cf7e64899a2bfce641404ad87 (127.0.98.2:35409)
I20260927 00:49:44.617684 14169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49199
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:49:44.627476 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34607
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:49:44.742367 14499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:49:44.742689 14499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:49:44.742779 14499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:49:44.746997 14499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:49:44.747105 14499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:49:44.747205 14499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:49:44.752714 14499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:49:44.753978 14499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:49:44.755172 14499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:49:44.765393 14505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:44.765486 14504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:44.765470 14507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:44.765841 14499 server_base.cc:1034] running on GCE node
I20260927 00:49:44.766299 14499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:49:44.767000 14499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:49:44.768191 14499 hybrid_clock.cc:648] HybridClock initialized: now 1790470184768180 us; error 39 us; skew 500 ppm
Sep 27 00:49:44 dist-test-slave-86rs krb5kdc[14124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:49:44.771178 14499 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:49:44.772179 14499 webserver.cc:460] Webserver started at http://127.0.98.3:36311/ using document root <none> and password file <none>
I20260927 00:49:44.772806 14499 fs_manager.cc:359] Metadata directory not provided
I20260927 00:49:44.772905 14499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:49:44.773164 14499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:49:44.775324 14499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/data/instance:
uuid: "03cbe42139b34d25a1f3f19c95c3c16c"
format_stamp: "Formatted at 2026-09-27 00:49:44 on dist-test-slave-86rs"
server_key: "10e9c1454c1926fd4ce32a7d12111343"
server_key_iv: "f061c013a5a5c0ca131e23d7ae533502"
server_key_version: "encryptionkey@0"
I20260927 00:49:44.775928 14499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/wal/instance:
uuid: "03cbe42139b34d25a1f3f19c95c3c16c"
format_stamp: "Formatted at 2026-09-27 00:49:44 on dist-test-slave-86rs"
server_key: "10e9c1454c1926fd4ce32a7d12111343"
server_key_iv: "f061c013a5a5c0ca131e23d7ae533502"
server_key_version: "encryptionkey@0"
I20260927 00:49:44.779836 14499 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260927 00:49:44.782584 14514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:44.783911 14499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:49:44.784081 14499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "03cbe42139b34d25a1f3f19c95c3c16c"
format_stamp: "Formatted at 2026-09-27 00:49:44 on dist-test-slave-86rs"
server_key: "10e9c1454c1926fd4ce32a7d12111343"
server_key_iv: "f061c013a5a5c0ca131e23d7ae533502"
server_key_version: "encryptionkey@0"
I20260927 00:49:44.784214 14499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:49:44.815961 14499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:49:44.819602 14499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:49:44.819854 14499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:49:44.820546 14499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:49:44.821448 14499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:49:44.821524 14499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:44.821596 14499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:49:44.821648 14499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:44.830730 14499 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42451
I20260927 00:49:44.830749 14627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42451 every 8 connection(s)
I20260927 00:49:44.831710 14499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:49:44.834787 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 14499
I20260927 00:49:44.834918 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:49:44.835196 392 external_mini_cluster.cc:1428] Setting key 3ac3eb6f66330cd766c90057383b3969
Sep 27 00:49:44 dist-test-slave-86rs krb5kdc[14124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:49:44.846197 14208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:44.833582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37023 (local address 127.0.98.62:34607)
0927 00:49:44.833757 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:44.833763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:44.834633 (+ 870us) server_negotiation.cc:407] Connection header received
0927 00:49:44.836402 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:44.836415 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:44.836473 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:44.836560 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:44.838159 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:44.838706 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:44.839913 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:44.840127 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:44.840207 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:44.842855 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:44.842859 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:44.842861 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:44.842885 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:44.844389 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:44.844825 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:44.844828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:44.844829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:44.844906 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:44.845247 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:44.845250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:44.845251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:44.845419 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:44.845493 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:44.845959 (+ 466us) server_negotiation.cc:299] Negotiation successful
0927 00:49:44.846070 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 00:49:44.847074 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:44.834007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.3:37023)
0927 00:49:44.834472 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:44.834518 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:44.836117 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:44.836787 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:44.836815 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:44.837405 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:49:44.838024 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:44.838040 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:44.838845 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:44.838849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:44.839765 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:44.839802 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:44.840363 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:44.840366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:44.840610 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:44.841172 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:49:44.841190 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:49:44.842702 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:49:44.844482 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:44.844486 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:44.844499 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:44.844743 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:44.844997 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:44.844999 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:44.845008 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:44.845170 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:44.845506 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:49:44.845512 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:49:44.845756 (+ 244us) client_negotiation.cc:769] Sending connection context
0927 00:49:44.845915 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 00:49:44.846186 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":118,"threads_started":1}
I20260927 00:49:44.848381 14628 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34607
I20260927 00:49:44.848651 14628 heartbeater.cc:464] Registering TS with master...
I20260927 00:49:44.849261 14628 heartbeater.cc:511] Master 127.0.98.62:34607 requested a full tablet report, sending...
I20260927 00:49:44.850369 14169 ts_manager.cc:194] Registered new tserver with Master: 03cbe42139b34d25a1f3f19c95c3c16c (127.0.98.3:42451)
I20260927 00:49:44.850944 14169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37023
I20260927 00:49:44.861696 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:49:44.871330 14208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:44.863806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47314 (local address 127.0.98.62:34607)
0927 00:49:44.863980 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:44.863993 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:44.864110 (+ 117us) server_negotiation.cc:407] Connection header received
0927 00:49:44.864213 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:44.864216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:44.864340 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:44.864422 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:44.865207 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:44.865760 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:44.866629 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:44.866847 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:44.866929 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:44.868425 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:44.868428 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:44.868431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:44.868456 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:44.869809 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:44.870200 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:44.870203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:44.870204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:44.870278 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:44.870501 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:44.870504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:44.870506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:44.870787 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:44.870891 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:44.871103 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 00:49:44.871206 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 00:49:44.874466 14169 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:49:44.877592 14169 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:49:44.893683 14643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:44.887835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46944 (local address 127.0.98.3:42451)
0927 00:49:44.888644 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:44.888658 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:44.888682 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:49:44.888758 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:44.888764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:44.888970 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:44.889115 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:49:44.889346 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:44.890507 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:44.892348 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:44.892917 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:44.893065 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:44.893230 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:44.893424 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 00:49:44.893523 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":144,"threads_started":1}
I20260927 00:49:44.893675 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:44.887151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42451 (local address 127.0.0.1:46944)
0927 00:49:44.888000 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:44.888114 (+ 114us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:44.888200 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:44.889105 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:44.889111 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:44.889130 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:49:44.889222 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:44.889228 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:44.890664 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:44.890668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:44.891952 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:44.891963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:44.893064 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:44.893067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:44.893191 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:44.893230 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 00:49:44.893388 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 00:49:44.893511 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":53,"thread_start_us":95,"threads_started":1}
I20260927 00:49:44.894179 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:44.886949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35409 (local address 127.0.0.1:37278)
0927 00:49:44.887375 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:44.887452 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:44.887594 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:44.888555 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:44.888561 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:44.888585 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:49:44.888783 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:44.888792 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:44.890282 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:44.890286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:44.891954 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:44.891963 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:44.893120 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:44.893124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:44.893212 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:44.893234 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:49:44.893322 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 00:49:44.893481 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":53,"thread_start_us":118,"threads_started":1}
I20260927 00:49:44.894207 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:44.888324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46220 (local address 127.0.98.1:43081)
0927 00:49:44.888664 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:44.888675 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:44.889004 (+ 329us) server_negotiation.cc:407] Connection header received
0927 00:49:44.889175 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:44.889180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:44.889364 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:44.889505 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:49:44.889749 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:44.890806 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:44.892155 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:44.892963 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:44.893091 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:44.893233 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:44.893747 (+ 514us) server_negotiation.cc:299] Negotiation successful
0927 00:49:44.894072 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":78,"threads_started":1}
I20260927 00:49:44.893679 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:44.887747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43081 (local address 127.0.0.1:46220)
0927 00:49:44.888896 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:44.888910 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:44.889005 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:44.889502 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:44.889505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:44.889523 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:49:44.889620 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:44.889629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:44.890927 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:44.890932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:44.891986 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:44.891996 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:44.893293 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:44.893300 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:44.893386 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:44.893404 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 00:49:44.893464 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 00:49:44.893531 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1091,"mutex_wait_us":8,"thread_start_us":96,"threads_started":1}
I20260927 00:49:44.894212 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:44.887132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37278 (local address 127.0.98.2:35409)
0927 00:49:44.887955 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:44.887969 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:44.888004 (+ 35us) server_negotiation.cc:407] Connection header received
0927 00:49:44.888089 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:44.888097 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:44.888348 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:44.888948 (+ 600us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:49:44.889139 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:44.890153 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:44.892095 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:44.892963 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:44.893603 (+ 640us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:44.893906 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:44.893956 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 00:49:44.894065 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":127,"threads_started":1}
I20260927 00:49:44.897162 14428 tablet_service.cc:1467] Processing CreateTablet for tablet 385d393c742948f68a83383537f4265f (DEFAULT_TABLE table=test-table [id=5d4aeb73e8e64bc8905eb84aff031ca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:49:44.897181 14294 tablet_service.cc:1467] Processing CreateTablet for tablet 385d393c742948f68a83383537f4265f (DEFAULT_TABLE table=test-table [id=5d4aeb73e8e64bc8905eb84aff031ca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:49:44.897181 14562 tablet_service.cc:1467] Processing CreateTablet for tablet 385d393c742948f68a83383537f4265f (DEFAULT_TABLE table=test-table [id=5d4aeb73e8e64bc8905eb84aff031ca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:49:44.898226 14428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 385d393c742948f68a83383537f4265f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:49:44.898234 14294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 385d393c742948f68a83383537f4265f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:49:44.898248 14562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 385d393c742948f68a83383537f4265f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:49:44.905295 14645 tablet_bootstrap.cc:492] T 385d393c742948f68a83383537f4265f P 06b92c2cf7e64899a2bfce641404ad87: Bootstrap starting.
I20260927 00:49:44.906800 14646 tablet_bootstrap.cc:492] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c: Bootstrap starting.
I20260927 00:49:44.906462 14647 tablet_bootstrap.cc:492] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c: Bootstrap starting.
I20260927 00:49:44.908663 14645 tablet_bootstrap.cc:654] T 385d393c742948f68a83383537f4265f P 06b92c2cf7e64899a2bfce641404ad87: Neither blocks nor log segments found. Creating new log.
I20260927 00:49:44.908959 14646 tablet_bootstrap.cc:654] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c: Neither blocks nor log segments found. Creating new log.
I20260927 00:49:44.909598 14645 log.cc:824] T 385d393c742948f68a83383537f4265f P 06b92c2cf7e64899a2bfce641404ad87: Log is configured to *not* fsync() on all Append() calls
I20260927 00:49:44.909737 14646 log.cc:824] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:49:44.910399 14647 tablet_bootstrap.cc:654] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c: Neither blocks nor log segments found. Creating new log.
I20260927 00:49:44.911669 14647 log.cc:824] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:49:44.911888 14646 tablet_bootstrap.cc:492] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c: No bootstrap required, opened a new log
I20260927 00:49:44.912118 14646 ts_tablet_manager.cc:1397] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:49:44.912128 14645 tablet_bootstrap.cc:492] T 385d393c742948f68a83383537f4265f P 06b92c2cf7e64899a2bfce641404ad87: No bootstrap required, opened a new log
I20260927 00:49:44.912356 14645 ts_tablet_manager.cc:1397] T 385d393c742948f68a83383537f4265f P 06b92c2cf7e64899a2bfce641404ad87: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 00:49:44.913643 14647 tablet_bootstrap.cc:492] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c: No bootstrap required, opened a new log
I20260927 00:49:44.913825 14647 ts_tablet_manager.cc:1397] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260927 00:49:44.915803 14646 raft_consensus.cc:348] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } }
I20260927 00:49:44.916110 14646 raft_consensus.cc:374] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:49:44.916169 14646 raft_consensus.cc:729] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5bfcca01792442087b4108656ab072c, State: Initialized, Role: FOLLOWER
I20260927 00:49:44.916296 14645 raft_consensus.cc:348] T 385d393c742948f68a83383537f4265f P 06b92c2cf7e64899a2bfce641404ad87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } }
I20260927 00:49:44.916599 14645 raft_consensus.cc:374] T 385d393c742948f68a83383537f4265f P 06b92c2cf7e64899a2bfce641404ad87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:49:44.916661 14645 raft_consensus.cc:729] T 385d393c742948f68a83383537f4265f P 06b92c2cf7e64899a2bfce641404ad87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06b92c2cf7e64899a2bfce641404ad87, State: Initialized, Role: FOLLOWER
I20260927 00:49:44.916751 14646 consensus_queue.cc:260] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } }
I20260927 00:49:44.917095 14645 consensus_queue.cc:260] T 385d393c742948f68a83383537f4265f P 06b92c2cf7e64899a2bfce641404ad87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } }
I20260927 00:49:44.917563 14360 heartbeater.cc:503] Master 127.0.98.62:34607 was elected leader, sending a full tablet report...
I20260927 00:49:44.917711 14647 raft_consensus.cc:348] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } }
I20260927 00:49:44.917906 14646 ts_tablet_manager.cc:1428] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:49:44.917968 14647 raft_consensus.cc:374] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:49:44.918017 14647 raft_consensus.cc:729] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03cbe42139b34d25a1f3f19c95c3c16c, State: Initialized, Role: FOLLOWER
I20260927 00:49:44.918076 14645 ts_tablet_manager.cc:1428] T 385d393c742948f68a83383537f4265f P 06b92c2cf7e64899a2bfce641404ad87: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:49:44.918394 14494 heartbeater.cc:503] Master 127.0.98.62:34607 was elected leader, sending a full tablet report...
I20260927 00:49:44.918859 14647 consensus_queue.cc:260] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } }
I20260927 00:49:44.920053 14647 ts_tablet_manager.cc:1428] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:49:44.920414 14628 heartbeater.cc:503] Master 127.0.98.62:34607 was elected leader, sending a full tablet report...
W20260927 00:49:45.082839 14629 tablet.cc:2369] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:49:45.095409 14495 tablet.cc:2369] T 385d393c742948f68a83383537f4265f P 06b92c2cf7e64899a2bfce641404ad87: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:49:45.112967 14361 tablet.cc:2369] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:49:45.164785 14653 raft_consensus.cc:482] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:49:45.165081 14653 raft_consensus.cc:504] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } }
I20260927 00:49:45.166334 14653 leader_election.cc:288] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06b92c2cf7e64899a2bfce641404ad87 (127.0.98.2:35409), e5bfcca01792442087b4108656ab072c (127.0.98.1:43081)
I20260927 00:49:45.170905 14652 raft_consensus.cc:482] T 385d393c742948f68a83383537f4265f P 06b92c2cf7e64899a2bfce641404ad87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:49:45.171154 14652 raft_consensus.cc:504] T 385d393c742948f68a83383537f4265f P 06b92c2cf7e64899a2bfce641404ad87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } }
I20260927 00:49:45.171748 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:45.166718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35409 (local address 127.0.98.3:42635)
0927 00:49:45.166883 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:45.166900 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:45.167042 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:45.167406 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:45.167410 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:45.167438 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:49:45.167615 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:45.167622 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.168724 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.168727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:45.170186 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:45.170197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.171199 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.171203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:45.171315 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:45.171405 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 00:49:45.171488 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 00:49:45.171603 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":112,"spinlock_wait_cycles":27392}
I20260927 00:49:45.171747 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:45.166791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42635 (local address 127.0.98.2:35409)
0927 00:49:45.166946 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:45.166951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:45.166973 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:49:45.167168 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:45.167172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:45.167224 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:45.167302 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:49:45.167736 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.168589 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.170339 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.170944 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.171070 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:45.171142 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:45.171570 (+ 428us) server_negotiation.cc:299] Negotiation successful
0927 00:49:45.171648 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 00:49:45.171789 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:45.167269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52641 (local address 127.0.98.1:43081)
0927 00:49:45.167416 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:45.167420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:45.167466 (+ 46us) server_negotiation.cc:407] Connection header received
0927 00:49:45.167622 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:45.167625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:45.167692 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:45.167771 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:49:45.168006 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.168741 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.170322 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.170998 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.171087 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:45.171142 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:45.171559 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 00:49:45.171648 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 00:49:45.172173 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:45.167045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43081 (local address 127.0.98.3:52641)
0927 00:49:45.167392 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:45.167405 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:45.167520 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:45.167794 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:45.167796 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:45.167813 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:49:45.167897 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:45.167901 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.168841 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.168844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:45.170186 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:45.170197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.171215 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.171218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:45.171372 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:45.171407 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 00:49:45.171488 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 00:49:45.172052 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":463,"spinlock_wait_cycles":3200,"thread_start_us":162,"threads_started":1}
I20260927 00:49:45.172703 14652 leader_election.cc:288] T 385d393c742948f68a83383537f4265f P 06b92c2cf7e64899a2bfce641404ad87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03cbe42139b34d25a1f3f19c95c3c16c (127.0.98.3:42451), e5bfcca01792442087b4108656ab072c (127.0.98.1:43081)
I20260927 00:49:45.172811 14448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385d393c742948f68a83383537f4265f" candidate_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06b92c2cf7e64899a2bfce641404ad87" is_pre_election: true
I20260927 00:49:45.172809 14314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385d393c742948f68a83383537f4265f" candidate_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5bfcca01792442087b4108656ab072c" is_pre_election: true
I20260927 00:49:45.173040 14448 raft_consensus.cc:2454] T 385d393c742948f68a83383537f4265f P 06b92c2cf7e64899a2bfce641404ad87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03cbe42139b34d25a1f3f19c95c3c16c in term 0.
I20260927 00:49:45.173120 14314 raft_consensus.cc:2454] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03cbe42139b34d25a1f3f19c95c3c16c in term 0.
I20260927 00:49:45.173636 14516 leader_election.cc:302] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03cbe42139b34d25a1f3f19c95c3c16c, e5bfcca01792442087b4108656ab072c; no voters:
I20260927 00:49:45.173982 14653 raft_consensus.cc:2781] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:49:45.174074 14653 raft_consensus.cc:482] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:49:45.174129 14653 raft_consensus.cc:3037] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:49:45.175233 14653 raft_consensus.cc:504] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } }
I20260927 00:49:45.175621 14653 leader_election.cc:288] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c [CANDIDATE]: Term 1 election: Requested vote from peers 06b92c2cf7e64899a2bfce641404ad87 (127.0.98.2:35409), e5bfcca01792442087b4108656ab072c (127.0.98.1:43081)
I20260927 00:49:45.176047 14314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385d393c742948f68a83383537f4265f" candidate_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5bfcca01792442087b4108656ab072c"
I20260927 00:49:45.176054 14448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385d393c742948f68a83383537f4265f" candidate_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06b92c2cf7e64899a2bfce641404ad87"
I20260927 00:49:45.176167 14448 raft_consensus.cc:3037] T 385d393c742948f68a83383537f4265f P 06b92c2cf7e64899a2bfce641404ad87 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:49:45.176175 14314 raft_consensus.cc:3037] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:49:45.177541 14448 raft_consensus.cc:2454] T 385d393c742948f68a83383537f4265f P 06b92c2cf7e64899a2bfce641404ad87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03cbe42139b34d25a1f3f19c95c3c16c in term 1.
I20260927 00:49:45.177537 14314 raft_consensus.cc:2454] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03cbe42139b34d25a1f3f19c95c3c16c in term 1.
I20260927 00:49:45.177863 14516 leader_election.cc:302] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03cbe42139b34d25a1f3f19c95c3c16c, e5bfcca01792442087b4108656ab072c; no voters:
I20260927 00:49:45.178088 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:45.173314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46109 (local address 127.0.98.1:43081)
0927 00:49:45.173493 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:45.173497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:45.173680 (+ 183us) server_negotiation.cc:407] Connection header received
0927 00:49:45.173846 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:45.173849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:45.173893 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:45.173938 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:49:45.174289 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.174996 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.176706 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.177318 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.177416 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:45.177521 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:45.177864 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 00:49:45.177956 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 00:49:45.178103 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:45.173014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43081 (local address 127.0.98.2:46109)
0927 00:49:45.173588 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:45.173601 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:45.173735 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:45.174052 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:45.174054 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:45.174067 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:49:45.174180 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:45.174186 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.175123 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.175126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:45.176566 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:45.176576 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.177622 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.177625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:45.177749 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:45.177773 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:49:45.177842 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 00:49:45.177926 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":98,"thread_start_us":77,"threads_started":1}
I20260927 00:49:45.178254 14643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:45.173200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44013 (local address 127.0.98.3:42451)
0927 00:49:45.173316 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:45.173321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:45.173539 (+ 218us) server_negotiation.cc:407] Connection header received
0927 00:49:45.173689 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:45.173695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:45.173748 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:45.173818 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:49:45.174189 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.174972 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.176756 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.177376 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.177483 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:45.177563 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:45.178060 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 00:49:45.178144 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:49:45.178236 14653 raft_consensus.cc:2781] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:49:45.178493 14653 raft_consensus.cc:686] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c [term 1 LEADER]: Becoming Leader. State: Replica: 03cbe42139b34d25a1f3f19c95c3c16c, State: Running, Role: LEADER
I20260927 00:49:45.178813 14655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:45.172873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42451 (local address 127.0.98.2:44013)
0927 00:49:45.173453 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:45.173469 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:45.173578 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:45.173921 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:45.173925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:45.173942 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:49:45.174069 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:45.174075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.175123 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.175126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:45.176616 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:45.176626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.177737 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.177744 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:45.177854 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:45.177879 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:49:45.178612 (+ 733us) client_negotiation.cc:240] Negotiation successful
0927 00:49:45.178685 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":98,"thread_start_us":85,"threads_started":1}
I20260927 00:49:45.178884 14653 consensus_queue.cc:237] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } }
I20260927 00:49:45.179033 14314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385d393c742948f68a83383537f4265f" candidate_uuid: "06b92c2cf7e64899a2bfce641404ad87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5bfcca01792442087b4108656ab072c" is_pre_election: true
I20260927 00:49:45.179196 14314 raft_consensus.cc:2379] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 06b92c2cf7e64899a2bfce641404ad87 in current term 1: Already voted for candidate 03cbe42139b34d25a1f3f19c95c3c16c in this term.
I20260927 00:49:45.179384 14582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385d393c742948f68a83383537f4265f" candidate_uuid: "06b92c2cf7e64899a2bfce641404ad87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" is_pre_election: true
I20260927 00:49:45.180018 14381 leader_election.cc:302] T 385d393c742948f68a83383537f4265f P 06b92c2cf7e64899a2bfce641404ad87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06b92c2cf7e64899a2bfce641404ad87; no voters: 03cbe42139b34d25a1f3f19c95c3c16c, e5bfcca01792442087b4108656ab072c
I20260927 00:49:45.180323 14652 raft_consensus.cc:2726] T 385d393c742948f68a83383537f4265f P 06b92c2cf7e64899a2bfce641404ad87 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 00:49:45.182545 14168 catalog_manager.cc:5495] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c reported cstate change: term changed from 0 to 1, leader changed from <none> to 03cbe42139b34d25a1f3f19c95c3c16c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } health_report { overall_health: UNKNOWN } } }
I20260927 00:49:45.261785 14643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:45.257920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46956 (local address 127.0.98.3:42451)
0927 00:49:45.258068 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:45.258074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:45.258088 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:49:45.258281 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:45.258284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:45.258344 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:45.258464 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:49:45.258739 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.259261 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.260164 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.260420 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.260537 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:45.260893 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:49:45.261360 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:49:45.261498 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:45.261551 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 00:49:45.261611 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:49:45.268260 14448 raft_consensus.cc:1261] T 385d393c742948f68a83383537f4265f P 06b92c2cf7e64899a2bfce641404ad87 [term 1 FOLLOWER]: Refusing update from remote peer 03cbe42139b34d25a1f3f19c95c3c16c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:49:45.268204 14314 raft_consensus.cc:1261] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c [term 1 FOLLOWER]: Refusing update from remote peer 03cbe42139b34d25a1f3f19c95c3c16c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:49:45.269030 14657 consensus_queue.cc:1035] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:49:45.269234 14653 consensus_queue.cc:1035] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c [LEADER]: Connected to new peer: Peer: permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:49:45.277671 14662 mvcc.cc:205] Tried to move back new op lower bound from 7333765878850711552 to 7333765878496694272. Current Snapshot: MvccSnapshot[applied={T|T < 7333765878850711552}]
I20260927 00:49:45.286646 14167 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:49:45.286823 14167 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:49:45.289578 14167 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b: Sending DeleteTablet for 3 replicas of tablet 385d393c742948f68a83383537f4265f
I20260927 00:49:45.290464 14562 tablet_service.cc:1514] Processing DeleteTablet for tablet 385d393c742948f68a83383537f4265f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:49:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46944
I20260927 00:49:45.290772 14294 tablet_service.cc:1514] Processing DeleteTablet for tablet 385d393c742948f68a83383537f4265f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:49:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46220
I20260927 00:49:45.291078 14428 tablet_service.cc:1514] Processing DeleteTablet for tablet 385d393c742948f68a83383537f4265f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:49:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37278
I20260927 00:49:45.291150 14670 tablet_replica.cc:330] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c: stopping tablet replica
I20260927 00:49:45.291504 14670 raft_consensus.cc:2229] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:49:45.291630 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 14231
I20260927 00:49:45.291556 14672 tablet_replica.cc:330] T 385d393c742948f68a83383537f4265f P 06b92c2cf7e64899a2bfce641404ad87: stopping tablet replica
I20260927 00:49:45.291949 14670 raft_consensus.cc:2258] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:49:45.292086 14672 raft_consensus.cc:2229] T 385d393c742948f68a83383537f4265f P 06b92c2cf7e64899a2bfce641404ad87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:49:45.292480 14672 raft_consensus.cc:2258] T 385d393c742948f68a83383537f4265f P 06b92c2cf7e64899a2bfce641404ad87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:49:45.293604 14670 ts_tablet_manager.cc:1905] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:49:45.294097 14672 ts_tablet_manager.cc:1905] T 385d393c742948f68a83383537f4265f P 06b92c2cf7e64899a2bfce641404ad87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:49:45.296435 14670 ts_tablet_manager.cc:1918] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:49:45.296553 14670 log.cc:1196] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/wal/wals/385d393c742948f68a83383537f4265f
I20260927 00:49:45.296867 14670 ts_tablet_manager.cc:1939] T 385d393c742948f68a83383537f4265f P 03cbe42139b34d25a1f3f19c95c3c16c: Deleting consensus metadata
I20260927 00:49:45.297763 14672 ts_tablet_manager.cc:1918] T 385d393c742948f68a83383537f4265f P 06b92c2cf7e64899a2bfce641404ad87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:49:45.297936 14672 log.cc:1196] T 385d393c742948f68a83383537f4265f P 06b92c2cf7e64899a2bfce641404ad87: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/wal/wals/385d393c742948f68a83383537f4265f
I20260927 00:49:45.298182 14153 catalog_manager.cc:4841] TS 03cbe42139b34d25a1f3f19c95c3c16c (127.0.98.3:42451): tablet 385d393c742948f68a83383537f4265f (table test-table [id=5d4aeb73e8e64bc8905eb84aff031ca7]) successfully deleted
I20260927 00:49:45.298724 14672 ts_tablet_manager.cc:1939] T 385d393c742948f68a83383537f4265f P 06b92c2cf7e64899a2bfce641404ad87: Deleting consensus metadata
I20260927 00:49:45.300122 14154 catalog_manager.cc:4841] TS 06b92c2cf7e64899a2bfce641404ad87 (127.0.98.2:35409): tablet 385d393c742948f68a83383537f4265f (table test-table [id=5d4aeb73e8e64bc8905eb84aff031ca7]) successfully deleted
W20260927 00:49:45.301195 14154 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43081 (error 108)
I20260927 00:49:45.301465 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 14365
I20260927 00:49:45.305640 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:45.305240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43081 (local address 127.0.0.1:46234)
0927 00:49:45.305411 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:45.305512 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 00:49:45.305830 14154 catalog_manager.cc:4584] TS e5bfcca01792442087b4108656ab072c (127.0.98.1:43081): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 385d393c742948f68a83383537f4265f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43081: connect: Connection refused (error 111)
I20260927 00:49:45.307718 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 14499
I20260927 00:49:45.313282 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 14137
I20260927 00:49:45.320031 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34607
--webserver_interface=127.0.98.62
--webserver_port=45967
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:49:45.435154 14674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:49:45.435532 14674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:49:45.435617 14674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:49:45.439960 14674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:49:45.440063 14674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:49:45.440100 14674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:49:45.440142 14674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:49:45.440181 14674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:49:45.445854 14674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:49:45.447103 14674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:49:45.448266 14674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:49:45.462960 14683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:45.462982 14681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:45.462960 14680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:45.463428 14674 server_base.cc:1034] running on GCE node
I20260927 00:49:45.464131 14674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:49:45.465461 14674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:49:45.466638 14674 hybrid_clock.cc:648] HybridClock initialized: now 1790470185466620 us; error 32 us; skew 500 ppm
Sep 27 00:49:45 dist-test-slave-86rs krb5kdc[14124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:49:45.469636 14674 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:49:45.470669 14674 webserver.cc:460] Webserver started at http://127.0.98.62:45967/ using document root <none> and password file <none>
I20260927 00:49:45.471279 14674 fs_manager.cc:359] Metadata directory not provided
I20260927 00:49:45.471390 14674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:49:45.475592 14674 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:49:45.478092 14690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:45.479468 14674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:49:45.479638 14674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "98ab283e0f91492083a657e1243d264b"
format_stamp: "Formatted at 2026-09-27 00:49:44 on dist-test-slave-86rs"
server_key: "dc0812c066da0be98532959f6ea74f55"
server_key_iv: "a9c0c0cce32783683fcd272a80019b63"
server_key_version: "encryptionkey@0"
I20260927 00:49:45.480345 14674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:49:45.498999 14674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:49:45.502084 14674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:49:45.502297 14674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:49:45.510111 14674 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34607
I20260927 00:49:45.510425 14742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34607 every 8 connection(s)
I20260927 00:49:45.511382 14674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:49:45.516203 14743 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:49:45.516918 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 14674
I20260927 00:49:45.517362 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43081
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34607
--webserver_port=35185
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:49:45.518285 14743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b: Bootstrap starting.
I20260927 00:49:45.535852 14743 log.cc:824] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b: Log is configured to *not* fsync() on all Append() calls
I20260927 00:49:45.544950 14743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:49:45.545401 14743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b: Bootstrap complete.
I20260927 00:49:45.549662 14743 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ab283e0f91492083a657e1243d264b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34607 } }
I20260927 00:49:45.550079 14743 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98ab283e0f91492083a657e1243d264b, State: Initialized, Role: FOLLOWER
I20260927 00:49:45.550611 14743 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ab283e0f91492083a657e1243d264b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34607 } }
I20260927 00:49:45.550735 14743 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:49:45.550832 14743 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:49:45.550948 14743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:49:45.553134 14743 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ab283e0f91492083a657e1243d264b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34607 } }
I20260927 00:49:45.553511 14743 leader_election.cc:302] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98ab283e0f91492083a657e1243d264b; no voters:
I20260927 00:49:45.553869 14743 leader_election.cc:288] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 00:49:45.553965 14747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [term 2 FOLLOWER]: Leader election won for term 2
I20260927 00:49:45.554176 14747 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [term 2 LEADER]: Becoming Leader. State: Replica: 98ab283e0f91492083a657e1243d264b, State: Running, Role: LEADER
I20260927 00:49:45.554486 14747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ab283e0f91492083a657e1243d264b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34607 } }
I20260927 00:49:45.555202 14743 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:49:45.555482 14747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98ab283e0f91492083a657e1243d264b. Latest consensus state: current_term: 2 leader_uuid: "98ab283e0f91492083a657e1243d264b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ab283e0f91492083a657e1243d264b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34607 } } }
I20260927 00:49:45.555609 14747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [sys.catalog]: This master's current role is: LEADER
I20260927 00:49:45.555972 14751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:49:45.555999 14748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "98ab283e0f91492083a657e1243d264b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ab283e0f91492083a657e1243d264b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34607 } } }
I20260927 00:49:45.556252 14748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [sys.catalog]: This master's current role is: LEADER
I20260927 00:49:45.562099 14751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:49:45.562767 14751 catalog_manager.cc:1259] Loaded cluster ID: ca38794ed4a543a188b45b3426c8cee2
I20260927 00:49:45.562834 14751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:49:45.564563 14751 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 00:49:45.569588 14751 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 00:49:45.575304 14751 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:49:45.576174 14751 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b: Loaded TSK: 0
I20260927 00:49:45.577474 14751 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:49:45.637486 14745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:49:45.637827 14745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:49:45.637903 14745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:49:45.642268 14745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:49:45.642377 14745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:49:45.642480 14745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:49:45.647938 14745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35185
--tserver_master_addrs=127.0.98.62:34607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:49:45.649106 14745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:49:45.650192 14745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:49:45.661125 14771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:45.661170 14773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:45.661324 14745 server_base.cc:1034] running on GCE node
W20260927 00:49:45.661131 14770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:45.662024 14745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:49:45.662858 14745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:49:45.664062 14745 hybrid_clock.cc:648] HybridClock initialized: now 1790470185664031 us; error 52 us; skew 500 ppm
Sep 27 00:49:45 dist-test-slave-86rs krb5kdc[14124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:49:45.667002 14745 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:49:45.668081 14745 webserver.cc:460] Webserver started at http://127.0.98.1:35185/ using document root <none> and password file <none>
I20260927 00:49:45.668720 14745 fs_manager.cc:359] Metadata directory not provided
I20260927 00:49:45.668799 14745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:49:45.673004 14745 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:49:45.675386 14780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:45.676844 14745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:49:45.676978 14745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "e5bfcca01792442087b4108656ab072c"
format_stamp: "Formatted at 2026-09-27 00:49:44 on dist-test-slave-86rs"
server_key: "67d708c3b58b8ffa4af2715c4812134f"
server_key_iv: "5eb5d584d11a602dec080be37068f7a2"
server_key_version: "encryptionkey@0"
I20260927 00:49:45.677565 14745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:49:45.718870 14745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:49:45.722132 14745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:49:45.722353 14745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:49:45.723001 14745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:49:45.724190 14787 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 00:49:45.728569 14745 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 00:49:45.728658 14745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 00:49:45.728761 14745 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 00:49:45.730508 14745 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 00:49:45.730587 14745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 00:49:45.730911 14787 tablet_bootstrap.cc:492] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c: Bootstrap starting.
I20260927 00:49:45.739864 14745 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43081
I20260927 00:49:45.739953 14894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43081 every 8 connection(s)
I20260927 00:49:45.740955 14745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:49:45.744706 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 14745
I20260927 00:49:45.745218 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35409
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34607
--webserver_port=33755
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:49:45.751914 14787 log.cc:824] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:49:45 dist-test-slave-86rs krb5kdc[14124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:49:45.761384 14787 tablet_bootstrap.cc:492] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:49:45.762250 14787 tablet_bootstrap.cc:492] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c: Bootstrap complete.
I20260927 00:49:45.762576 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:45.743269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.1:60825)
0927 00:49:45.744368 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:45.744459 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:45.745675 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:45.747300 (+ 1625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:45.747311 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:45.747855 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:49:45.748432 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:45.748449 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.750508 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.750510 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:45.751443 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:45.751453 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.752306 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.752310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:45.752501 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:45.753168 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:49:45.753190 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:49:45.755838 (+ 2648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:49:45.758586 (+ 2748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:45.758594 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:45.758610 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:45.759039 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:45.759493 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:45.759499 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:45.759503 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:45.759700 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:45.760489 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:49:45.760495 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:49:45.760817 (+ 322us) client_negotiation.cc:769] Sending connection context
0927 00:49:45.761026 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 00:49:45.761376 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"thread_start_us":129,"threads_started":1}
I20260927 00:49:45.762605 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:45.743340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60825 (local address 127.0.98.62:34607)
0927 00:49:45.743784 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:45.743810 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:45.744608 (+ 798us) server_negotiation.cc:407] Connection header received
0927 00:49:45.746223 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:45.746247 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:45.746790 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:45.747188 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:45.748621 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.750352 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.751598 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.752123 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.752344 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:45.756084 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:45.756091 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:45.756109 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:45.756146 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:45.758442 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:45.759203 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:45.759210 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:45.759216 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:45.759376 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:45.759826 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:45.759829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:45.759831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:45.760363 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:45.760500 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:45.761206 (+ 706us) server_negotiation.cc:299] Negotiation successful
0927 00:49:45.761648 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":128,"threads_started":1}
I20260927 00:49:45.763648 14787 ts_tablet_manager.cc:1397] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c: Time spent bootstrapping tablet: real 0.033s user 0.030s sys 0.000s
I20260927 00:49:45.765602 14895 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34607
I20260927 00:49:45.765851 14895 heartbeater.cc:464] Registering TS with master...
I20260927 00:49:45.766419 14895 heartbeater.cc:511] Master 127.0.98.62:34607 requested a full tablet report, sending...
I20260927 00:49:45.766582 14787 raft_consensus.cc:348] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } }
I20260927 00:49:45.767031 14787 raft_consensus.cc:729] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5bfcca01792442087b4108656ab072c, State: Initialized, Role: FOLLOWER
I20260927 00:49:45.767484 14787 consensus_queue.cc:260] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } }
I20260927 00:49:45.768602 14787 ts_tablet_manager.cc:1428] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:49:45.770498 14703 ts_manager.cc:194] Registered new tserver with Master: e5bfcca01792442087b4108656ab072c (127.0.98.1:43081)
I20260927 00:49:45.774209 14703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60825
I20260927 00:49:45.777027 14895 heartbeater.cc:503] Master 127.0.98.62:34607 was elected leader, sending a full tablet report...
Sep 27 00:49:45 dist-test-slave-86rs krb5kdc[14124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:49:45.785279 14902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:45.773570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43081 (local address 127.0.0.1:46244)
0927 00:49:45.774363 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:45.774415 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:45.774572 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:45.775244 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:45.775250 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:45.775699 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:49:45.775843 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:45.775852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.777247 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.777250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:45.778096 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:45.778106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.778706 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.778727 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:45.778853 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:45.779515 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:49:45.779527 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:49:45.781144 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:49:45.783317 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:45.783327 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:45.783332 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:45.783575 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:45.783888 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:45.783892 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:45.783894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:45.783992 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:45.784562 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:49:45.784568 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:49:45.784704 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 00:49:45.784844 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 00:49:45.785080 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":107,"threads_started":1}
I20260927 00:49:45.785279 14903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:45.773678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46244 (local address 127.0.98.1:43081)
0927 00:49:45.774076 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:45.774088 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:45.774531 (+ 443us) server_negotiation.cc:407] Connection header received
0927 00:49:45.774736 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:45.774743 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:45.774955 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:45.775137 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:45.775998 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.777088 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.778225 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.778580 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.778705 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:45.781308 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:45.781317 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:45.781328 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:45.781352 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:45.783183 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:45.783705 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:45.783710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:45.783712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:45.783789 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:45.784084 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:45.784089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:45.784091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:45.784445 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:45.784588 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:45.784939 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 00:49:45.785111 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":91,"threads_started":1}
I20260927 00:49:45.786525 14817 tablet_service.cc:1514] Processing DeleteTablet for tablet 385d393c742948f68a83383537f4265f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:49:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46244
I20260927 00:49:45.787595 14904 tablet_replica.cc:330] stopping tablet replica
I20260927 00:49:45.787925 14904 raft_consensus.cc:2229] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:49:45.788209 14904 raft_consensus.cc:2258] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:49:45.789271 14904 ts_tablet_manager.cc:1905] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:49:45.793165 14904 ts_tablet_manager.cc:1918] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:49:45.793299 14904 log.cc:1196] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/wal/wals/385d393c742948f68a83383537f4265f
I20260927 00:49:45.793578 14904 ts_tablet_manager.cc:1939] T 385d393c742948f68a83383537f4265f P e5bfcca01792442087b4108656ab072c: Deleting consensus metadata
I20260927 00:49:45.795442 14692 catalog_manager.cc:4841] TS e5bfcca01792442087b4108656ab072c (127.0.98.1:43081): tablet 385d393c742948f68a83383537f4265f (table test-table [id=5d4aeb73e8e64bc8905eb84aff031ca7]) successfully deleted
W20260927 00:49:45.874059 14899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:49:45.874426 14899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:49:45.874513 14899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:49:45.878731 14899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:49:45.878808 14899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:49:45.878870 14899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:49:45.884919 14899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35409
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33755
--tserver_master_addrs=127.0.98.62:34607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:49:45.886157 14899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:49:45.887262 14899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:49:45.897616 14912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:45.897701 14899 server_base.cc:1034] running on GCE node
W20260927 00:49:45.897637 14910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:45.897637 14909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:45.898422 14899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:49:45.899114 14899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:49:45.900338 14899 hybrid_clock.cc:648] HybridClock initialized: now 1790470185900299 us; error 54 us; skew 500 ppm
Sep 27 00:49:45 dist-test-slave-86rs krb5kdc[14124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:49:45.903714 14899 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:49:45.904862 14899 webserver.cc:460] Webserver started at http://127.0.98.2:33755/ using document root <none> and password file <none>
I20260927 00:49:45.905508 14899 fs_manager.cc:359] Metadata directory not provided
I20260927 00:49:45.905589 14899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:49:45.909716 14899 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:49:45.911954 14919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:45.913354 14899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:49:45.913501 14899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "06b92c2cf7e64899a2bfce641404ad87"
format_stamp: "Formatted at 2026-09-27 00:49:44 on dist-test-slave-86rs"
server_key: "bf15d5a1b31ea7bddf7f3df1fdf6d7d2"
server_key_iv: "c7b397315648031aa244b245ca9eafed"
server_key_version: "encryptionkey@0"
I20260927 00:49:45.914091 14899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:49:45.935612 14899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:49:45.939229 14899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:49:45.939462 14899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:49:45.940112 14899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:49:45.941038 14899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:49:45.941123 14899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:45.941198 14899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:49:45.941246 14899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:45.950778 14899 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35409
I20260927 00:49:45.950800 15032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35409 every 8 connection(s)
I20260927 00:49:45.951915 14899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:49:45.955941 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 14899
I20260927 00:49:45.956498 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42451
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34607
--webserver_port=36311
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:49:45 dist-test-slave-86rs krb5kdc[14124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:49:45.969725 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:45.953915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55179 (local address 127.0.98.62:34607)
0927 00:49:45.954097 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:45.954102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:45.954919 (+ 817us) server_negotiation.cc:407] Connection header received
0927 00:49:45.956535 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:45.956537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:45.956626 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:45.956734 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:45.958387 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.959301 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.961253 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.961493 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.961600 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:45.964999 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:45.965002 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:45.965004 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:45.965038 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:45.966618 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:45.967226 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:45.967230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:45.967231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:45.967316 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:45.967958 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:45.967961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:45.967963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:45.968289 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:45.968393 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:45.969270 (+ 877us) server_negotiation.cc:299] Negotiation successful
0927 00:49:45.969480 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 00:49:45.970728 15035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:45.954307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.2:55179)
0927 00:49:45.954773 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:45.954819 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:45.956230 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:45.956953 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:45.956966 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:45.957525 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:49:45.958221 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:45.958238 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.959473 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.959480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:45.961115 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:45.961131 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:45.961690 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:45.961697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:45.961966 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:45.962729 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:49:45.962774 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:49:45.964778 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:49:45.966764 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:45.966771 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:45.966790 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:45.967099 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:45.967414 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:45.967417 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:45.967419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:45.967580 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:45.968419 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:49:45.968428 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:49:45.968879 (+ 451us) client_negotiation.cc:769] Sending connection context
0927 00:49:45.969136 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 00:49:45.969519 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":120,"threads_started":1}
I20260927 00:49:45.972082 15033 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34607
I20260927 00:49:45.972394 15033 heartbeater.cc:464] Registering TS with master...
I20260927 00:49:45.972939 15033 heartbeater.cc:511] Master 127.0.98.62:34607 requested a full tablet report, sending...
I20260927 00:49:45.974086 14703 ts_manager.cc:194] Registered new tserver with Master: 06b92c2cf7e64899a2bfce641404ad87 (127.0.98.2:35409)
I20260927 00:49:45.974956 14703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55179
W20260927 00:49:46.077085 15036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:49:46.077384 15036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:49:46.077438 15036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:49:46.081650 15036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:49:46.081725 15036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:49:46.081789 15036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:49:46.087844 15036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36311
--tserver_master_addrs=127.0.98.62:34607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:49:46.089093 15036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:49:46.090138 15036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:49:46.100329 15044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:46.100329 15041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:46.100356 15042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:46.100649 15036 server_base.cc:1034] running on GCE node
I20260927 00:49:46.101194 15036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:49:46.101835 15036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:49:46.103060 15036 hybrid_clock.cc:648] HybridClock initialized: now 1790470186103026 us; error 57 us; skew 500 ppm
Sep 27 00:49:46 dist-test-slave-86rs krb5kdc[14124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:49:46.106062 15036 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:49:46.107044 15036 webserver.cc:460] Webserver started at http://127.0.98.3:36311/ using document root <none> and password file <none>
I20260927 00:49:46.107650 15036 fs_manager.cc:359] Metadata directory not provided
I20260927 00:49:46.107767 15036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:49:46.112062 15036 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:49:46.114598 15051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:46.116117 15036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:49:46.116307 15036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "03cbe42139b34d25a1f3f19c95c3c16c"
format_stamp: "Formatted at 2026-09-27 00:49:44 on dist-test-slave-86rs"
server_key: "10e9c1454c1926fd4ce32a7d12111343"
server_key_iv: "f061c013a5a5c0ca131e23d7ae533502"
server_key_version: "encryptionkey@0"
I20260927 00:49:46.116951 15036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:49:46.131042 15036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:49:46.134296 15036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:49:46.134460 15036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:49:46.135036 15036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:49:46.135861 15036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:49:46.135908 15036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:46.135951 15036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:49:46.135985 15036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:46.145458 15036 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42451
I20260927 00:49:46.145483 15164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42451 every 8 connection(s)
I20260927 00:49:46.146551 15036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:49:46.154711 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 15036
Sep 27 00:49:46 dist-test-slave-86rs krb5kdc[14124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:49:46.163622 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:46.148689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47053 (local address 127.0.98.62:34607)
0927 00:49:46.148869 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:46.148873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:46.149729 (+ 856us) server_negotiation.cc:407] Connection header received
0927 00:49:46.151100 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:46.151103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:46.151155 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:46.151241 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:46.152654 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:46.153205 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:46.154586 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:46.154878 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:46.155008 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:46.158606 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:46.158609 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:46.158611 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:46.158637 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:46.160699 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:46.161292 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:46.161296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:46.161298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:46.161399 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:46.161847 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:46.161852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:46.161855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:46.162138 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:46.162285 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:46.163176 (+ 891us) server_negotiation.cc:299] Negotiation successful
0927 00:49:46.163372 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 00:49:46.164705 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:46.149088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.3:47053)
0927 00:49:46.149570 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:46.149613 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:46.150874 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:46.151449 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:46.151457 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:46.151856 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:49:46.152485 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:46.152503 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:46.153359 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:46.153362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:46.154383 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:46.154398 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:46.155074 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:46.155079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:46.155309 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:46.156060 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:49:46.156089 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:49:46.158335 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:49:46.160847 (+ 2512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:46.160854 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:46.160871 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:46.161179 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:46.161506 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:46.161510 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:46.161512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:46.161742 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:46.162314 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:49:46.162323 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:49:46.162780 (+ 457us) client_negotiation.cc:769] Sending connection context
0927 00:49:46.163086 (+ 306us) client_negotiation.cc:240] Negotiation successful
0927 00:49:46.163521 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":124,"threads_started":1}
I20260927 00:49:46.165283 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:46.155584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47326 (local address 127.0.98.62:34607)
0927 00:49:46.156047 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:46.156050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:46.156143 (+ 93us) server_negotiation.cc:407] Connection header received
0927 00:49:46.156329 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:46.156332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:46.156391 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:46.156484 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:46.157429 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:46.158170 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:46.159152 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:46.159574 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:46.159703 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:46.161369 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:46.161372 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:46.161374 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:46.161398 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:46.163360 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:46.163919 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:46.163925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:46.163927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:46.164025 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:46.164365 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:46.164368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:46.164370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:46.164568 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:46.164796 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:46.164984 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 00:49:46.165118 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":139,"threads_started":1}
I20260927 00:49:46.166345 15165 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34607
I20260927 00:49:46.166677 15165 heartbeater.cc:464] Registering TS with master...
I20260927 00:49:46.167277 15165 heartbeater.cc:511] Master 127.0.98.62:34607 requested a full tablet report, sending...
I20260927 00:49:46.168371 14706 ts_manager.cc:194] Registered new tserver with Master: 03cbe42139b34d25a1f3f19c95c3c16c (127.0.98.3:42451)
I20260927 00:49:46.169003 14706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47053
I20260927 00:49:46.179643 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:49:46.185439 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:46.180411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47332 (local address 127.0.98.62:34607)
0927 00:49:46.180554 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:46.180557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:46.180845 (+ 288us) server_negotiation.cc:407] Connection header received
0927 00:49:46.181141 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:46.181147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:46.181336 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:46.181473 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:49:46.181890 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:46.182461 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:46.183600 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:46.183852 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:46.184002 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:46.184419 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:49:46.184992 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:49:46.185125 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:46.185212 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 00:49:46.185285 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:49:46.186228 14706 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:47332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:49:46.189625 14706 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:49:46.199443 14817 tablet_service.cc:1467] Processing CreateTablet for tablet dc97813f83554e028981568de9f7c688 (DEFAULT_TABLE table=test-table [id=dd2c86273d7240d08a596d424070e21f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:49:46.200178 14817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc97813f83554e028981568de9f7c688. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:49:46.203159 14787 tablet_bootstrap.cc:492] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c: Bootstrap starting.
I20260927 00:49:46.204006 14902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:46.197496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42451 (local address 127.0.0.1:46964)
0927 00:49:46.197633 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:46.197649 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:46.197732 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:46.198810 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:46.198813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:46.198832 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:49:46.198981 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:46.198988 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:46.201079 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:46.201082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:46.202367 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:46.202375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:46.203525 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:46.203529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:46.203632 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:46.203693 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 00:49:46.203753 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 00:49:46.203849 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":71}
I20260927 00:49:46.204088 14787 tablet_bootstrap.cc:654] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c: Neither blocks nor log segments found. Creating new log.
I20260927 00:49:46.204329 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:46.198355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35409 (local address 127.0.0.1:37288)
0927 00:49:46.198583 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:46.198597 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:46.198697 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:46.199731 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:46.199734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:46.199750 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:49:46.199852 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:46.199858 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:46.201027 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:46.201033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:46.202623 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:46.202630 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:46.203838 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:46.203841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:46.203933 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:46.203970 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 00:49:46.204051 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 00:49:46.204148 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":31,"thread_start_us":97,"threads_started":1}
I20260927 00:49:46.204692 15174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:46.197873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46964 (local address 127.0.98.3:42451)
0927 00:49:46.198241 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:46.198251 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:46.198272 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:49:46.198340 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:46.198345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:46.198612 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:46.199294 (+ 682us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:49:46.199853 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:46.200963 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:46.202508 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:46.203375 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:46.204199 (+ 824us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:46.204378 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:46.204440 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 00:49:46.204550 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":159,"threads_started":1}
I20260927 00:49:46.204891 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:46.198556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37288 (local address 127.0.98.2:35409)
0927 00:49:46.199101 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:46.199113 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:46.199136 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:49:46.199235 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:46.199240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:46.199450 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:46.199614 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:49:46.200073 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:46.200887 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:46.202750 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:46.203707 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:46.203811 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:46.203956 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:46.204631 (+ 675us) server_negotiation.cc:299] Negotiation successful
0927 00:49:46.204748 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":123,"threads_started":1}
I20260927 00:49:46.205786 14787 tablet_bootstrap.cc:492] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c: No bootstrap required, opened a new log
I20260927 00:49:46.205875 14787 ts_tablet_manager.cc:1397] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:49:46.206321 14787 raft_consensus.cc:348] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } } peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } }
I20260927 00:49:46.206436 14787 raft_consensus.cc:374] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:49:46.206471 14787 raft_consensus.cc:729] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5bfcca01792442087b4108656ab072c, State: Initialized, Role: FOLLOWER
I20260927 00:49:46.206602 14787 consensus_queue.cc:260] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } } peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } }
I20260927 00:49:46.206874 14787 ts_tablet_manager.cc:1428] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:49:46.207844 15099 tablet_service.cc:1467] Processing CreateTablet for tablet dc97813f83554e028981568de9f7c688 (DEFAULT_TABLE table=test-table [id=dd2c86273d7240d08a596d424070e21f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:49:46.207846 14967 tablet_service.cc:1467] Processing CreateTablet for tablet dc97813f83554e028981568de9f7c688 (DEFAULT_TABLE table=test-table [id=dd2c86273d7240d08a596d424070e21f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:49:46.208925 14967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc97813f83554e028981568de9f7c688. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:49:46.208956 15099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc97813f83554e028981568de9f7c688. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:49:46.216385 15178 tablet_bootstrap.cc:492] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87: Bootstrap starting.
I20260927 00:49:46.216385 15179 tablet_bootstrap.cc:492] T dc97813f83554e028981568de9f7c688 P 03cbe42139b34d25a1f3f19c95c3c16c: Bootstrap starting.
I20260927 00:49:46.219019 15179 tablet_bootstrap.cc:654] T dc97813f83554e028981568de9f7c688 P 03cbe42139b34d25a1f3f19c95c3c16c: Neither blocks nor log segments found. Creating new log.
I20260927 00:49:46.219399 15178 tablet_bootstrap.cc:654] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87: Neither blocks nor log segments found. Creating new log.
I20260927 00:49:46.220127 15179 log.cc:824] T dc97813f83554e028981568de9f7c688 P 03cbe42139b34d25a1f3f19c95c3c16c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:49:46.220127 15178 log.cc:824] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87: Log is configured to *not* fsync() on all Append() calls
I20260927 00:49:46.222186 15179 tablet_bootstrap.cc:492] T dc97813f83554e028981568de9f7c688 P 03cbe42139b34d25a1f3f19c95c3c16c: No bootstrap required, opened a new log
I20260927 00:49:46.222184 15178 tablet_bootstrap.cc:492] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87: No bootstrap required, opened a new log
I20260927 00:49:46.222410 15179 ts_tablet_manager.cc:1397] T dc97813f83554e028981568de9f7c688 P 03cbe42139b34d25a1f3f19c95c3c16c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:49:46.222405 15178 ts_tablet_manager.cc:1397] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:49:46.225777 15178 raft_consensus.cc:348] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } } peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } }
I20260927 00:49:46.225777 15179 raft_consensus.cc:348] T dc97813f83554e028981568de9f7c688 P 03cbe42139b34d25a1f3f19c95c3c16c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } } peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } }
I20260927 00:49:46.226080 15178 raft_consensus.cc:374] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:49:46.226073 15179 raft_consensus.cc:374] T dc97813f83554e028981568de9f7c688 P 03cbe42139b34d25a1f3f19c95c3c16c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:49:46.226159 15179 raft_consensus.cc:729] T dc97813f83554e028981568de9f7c688 P 03cbe42139b34d25a1f3f19c95c3c16c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03cbe42139b34d25a1f3f19c95c3c16c, State: Initialized, Role: FOLLOWER
I20260927 00:49:46.226161 15178 raft_consensus.cc:729] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06b92c2cf7e64899a2bfce641404ad87, State: Initialized, Role: FOLLOWER
I20260927 00:49:46.226639 15178 consensus_queue.cc:260] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } } peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } }
I20260927 00:49:46.226639 15179 consensus_queue.cc:260] T dc97813f83554e028981568de9f7c688 P 03cbe42139b34d25a1f3f19c95c3c16c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } } peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } }
I20260927 00:49:46.227365 15165 heartbeater.cc:503] Master 127.0.98.62:34607 was elected leader, sending a full tablet report...
I20260927 00:49:46.227370 15033 heartbeater.cc:503] Master 127.0.98.62:34607 was elected leader, sending a full tablet report...
I20260927 00:49:46.227782 15179 ts_tablet_manager.cc:1428] T dc97813f83554e028981568de9f7c688 P 03cbe42139b34d25a1f3f19c95c3c16c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:49:46.228775 15178 ts_tablet_manager.cc:1428] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260927 00:49:46.242405 14896 tablet.cc:2369] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:49:46.330376 15182 raft_consensus.cc:482] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:49:46.330590 15182 raft_consensus.cc:504] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } } peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } }
I20260927 00:49:46.331753 15182 leader_election.cc:288] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03cbe42139b34d25a1f3f19c95c3c16c (127.0.98.3:42451), e5bfcca01792442087b4108656ab072c (127.0.98.1:43081)
I20260927 00:49:46.335798 15184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:46.332170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47585 (local address 127.0.98.1:43081)
0927 00:49:46.332512 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:46.332517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:46.332541 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:49:46.332590 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:46.332592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:46.332646 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:46.332765 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:49:46.333057 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:46.333576 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:46.334764 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:46.335228 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:46.335357 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:46.335543 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:46.335583 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 00:49:46.335655 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":133,"threads_started":1}
I20260927 00:49:46.335798 15035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:46.332075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43081 (local address 127.0.98.2:47585)
0927 00:49:46.332222 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:46.332257 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:46.332370 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:46.332788 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:46.332791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:46.332809 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:49:46.332934 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:46.332940 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:46.333691 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:46.333698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:46.334629 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:46.334638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:46.335357 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:46.335361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:46.335474 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:46.335496 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:49:46.335565 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 00:49:46.335647 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":60}
I20260927 00:49:46.335914 15174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:46.332316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34571 (local address 127.0.98.3:42451)
0927 00:49:46.332446 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:46.332451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:46.332732 (+ 281us) server_negotiation.cc:407] Connection header received
0927 00:49:46.332892 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:46.332895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:46.332950 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:46.333043 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:49:46.333297 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:46.333815 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:46.334895 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:46.335312 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:46.335443 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:46.335509 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:46.335676 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 00:49:46.335747 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:49:46.335973 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:46.332075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42451 (local address 127.0.98.2:34571)
0927 00:49:46.332629 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:46.332641 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:46.332774 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:46.333055 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:46.333057 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:46.333073 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:49:46.333194 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:46.333199 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:46.333947 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:46.333951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:46.334771 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:46.334779 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:46.335438 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:46.335441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:46.335546 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:46.335574 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 00:49:46.335661 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 00:49:46.335731 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":136,"thread_start_us":110,"threads_started":1}
I20260927 00:49:46.336733 15119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc97813f83554e028981568de9f7c688" candidate_uuid: "06b92c2cf7e64899a2bfce641404ad87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" is_pre_election: true
I20260927 00:49:46.336716 14838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc97813f83554e028981568de9f7c688" candidate_uuid: "06b92c2cf7e64899a2bfce641404ad87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5bfcca01792442087b4108656ab072c" is_pre_election: true
I20260927 00:49:46.337044 14838 raft_consensus.cc:2454] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06b92c2cf7e64899a2bfce641404ad87 in term 0.
I20260927 00:49:46.337055 15119 raft_consensus.cc:2454] T dc97813f83554e028981568de9f7c688 P 03cbe42139b34d25a1f3f19c95c3c16c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06b92c2cf7e64899a2bfce641404ad87 in term 0.
I20260927 00:49:46.337555 14921 leader_election.cc:302] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06b92c2cf7e64899a2bfce641404ad87, e5bfcca01792442087b4108656ab072c; no voters:
I20260927 00:49:46.337831 15182 raft_consensus.cc:2781] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:49:46.337921 15182 raft_consensus.cc:482] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:49:46.337966 15182 raft_consensus.cc:3037] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:49:46.339095 15182 raft_consensus.cc:504] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } } peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } }
I20260927 00:49:46.339454 15182 leader_election.cc:288] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87 [CANDIDATE]: Term 1 election: Requested vote from peers 03cbe42139b34d25a1f3f19c95c3c16c (127.0.98.3:42451), e5bfcca01792442087b4108656ab072c (127.0.98.1:43081)
I20260927 00:49:46.339845 15119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc97813f83554e028981568de9f7c688" candidate_uuid: "06b92c2cf7e64899a2bfce641404ad87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03cbe42139b34d25a1f3f19c95c3c16c"
I20260927 00:49:46.339859 14838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc97813f83554e028981568de9f7c688" candidate_uuid: "06b92c2cf7e64899a2bfce641404ad87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5bfcca01792442087b4108656ab072c"
I20260927 00:49:46.339973 15119 raft_consensus.cc:3037] T dc97813f83554e028981568de9f7c688 P 03cbe42139b34d25a1f3f19c95c3c16c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:49:46.340018 14838 raft_consensus.cc:3037] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:49:46.341137 15119 raft_consensus.cc:2454] T dc97813f83554e028981568de9f7c688 P 03cbe42139b34d25a1f3f19c95c3c16c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06b92c2cf7e64899a2bfce641404ad87 in term 1.
I20260927 00:49:46.341137 14838 raft_consensus.cc:2454] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06b92c2cf7e64899a2bfce641404ad87 in term 1.
I20260927 00:49:46.341478 14920 leader_election.cc:302] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 03cbe42139b34d25a1f3f19c95c3c16c, 06b92c2cf7e64899a2bfce641404ad87, e5bfcca01792442087b4108656ab072c; no voters:
I20260927 00:49:46.341737 15182 raft_consensus.cc:2781] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:49:46.341950 15182 raft_consensus.cc:686] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87 [term 1 LEADER]: Becoming Leader. State: Replica: 06b92c2cf7e64899a2bfce641404ad87, State: Running, Role: LEADER
I20260927 00:49:46.342327 15182 consensus_queue.cc:237] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } } peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } }
I20260927 00:49:46.346267 14707 catalog_manager.cc:5495] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06b92c2cf7e64899a2bfce641404ad87 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "06b92c2cf7e64899a2bfce641404ad87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } health_report { overall_health: UNKNOWN } } }
I20260927 00:49:46.361166 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:46.357793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37292 (local address 127.0.98.2:35409)
0927 00:49:46.357907 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:46.357911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:46.357924 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:49:46.358003 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:46.358005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:46.358062 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:46.358151 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:49:46.358346 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:46.358848 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:46.359610 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:46.359855 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:46.359943 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:46.360203 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:49:46.360737 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:49:46.360838 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:46.360906 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 00:49:46.360959 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 00:49:46.367471 14838 raft_consensus.cc:1261] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [term 1 FOLLOWER]: Refusing update from remote peer 06b92c2cf7e64899a2bfce641404ad87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:49:46.367470 15119 raft_consensus.cc:1261] T dc97813f83554e028981568de9f7c688 P 03cbe42139b34d25a1f3f19c95c3c16c [term 1 FOLLOWER]: Refusing update from remote peer 06b92c2cf7e64899a2bfce641404ad87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:49:46.368276 15186 consensus_queue.cc:1035] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:49:46.368613 15182 consensus_queue.cc:1035] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:49:46.376611 15191 mvcc.cc:205] Tried to move back new op lower bound from 7333765883353276416 to 7333765883261050880. Current Snapshot: MvccSnapshot[applied={T|T < 7333765883353276416}]
I20260927 00:49:46.377735 15190 mvcc.cc:205] Tried to move back new op lower bound from 7333765883353276416 to 7333765883261050880. Current Snapshot: MvccSnapshot[applied={T|T < 7333765883353276416}]
I20260927 00:49:46.384059 14703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:47332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:49:46.384294 14703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:47332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:49:46.387015 14703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b: Sending DeleteTablet for 3 replicas of tablet dc97813f83554e028981568de9f7c688
I20260927 00:49:46.387794 14817 tablet_service.cc:1514] Processing DeleteTablet for tablet dc97813f83554e028981568de9f7c688 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:49:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46244
I20260927 00:49:46.387812 15099 tablet_service.cc:1514] Processing DeleteTablet for tablet dc97813f83554e028981568de9f7c688 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:49:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46964
I20260927 00:49:46.387856 14967 tablet_service.cc:1514] Processing DeleteTablet for tablet dc97813f83554e028981568de9f7c688 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:49:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37288
I20260927 00:49:46.388362 15200 tablet_replica.cc:330] T dc97813f83554e028981568de9f7c688 P 03cbe42139b34d25a1f3f19c95c3c16c: stopping tablet replica
I20260927 00:49:46.388545 15201 tablet_replica.cc:330] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87: stopping tablet replica
I20260927 00:49:46.388572 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 14745
I20260927 00:49:46.389360 15200 raft_consensus.cc:2229] T dc97813f83554e028981568de9f7c688 P 03cbe42139b34d25a1f3f19c95c3c16c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:49:46.389416 15201 raft_consensus.cc:2229] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:49:46.389746 15200 raft_consensus.cc:2258] T dc97813f83554e028981568de9f7c688 P 03cbe42139b34d25a1f3f19c95c3c16c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:49:46.390055 15201 raft_consensus.cc:2258] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:49:46.391340 15200 ts_tablet_manager.cc:1905] T dc97813f83554e028981568de9f7c688 P 03cbe42139b34d25a1f3f19c95c3c16c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:49:46.391714 15201 ts_tablet_manager.cc:1905] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:49:46.394255 15200 ts_tablet_manager.cc:1918] T dc97813f83554e028981568de9f7c688 P 03cbe42139b34d25a1f3f19c95c3c16c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:49:46.394377 15200 log.cc:1196] T dc97813f83554e028981568de9f7c688 P 03cbe42139b34d25a1f3f19c95c3c16c: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/wal/wals/dc97813f83554e028981568de9f7c688
I20260927 00:49:46.394743 15200 ts_tablet_manager.cc:1939] T dc97813f83554e028981568de9f7c688 P 03cbe42139b34d25a1f3f19c95c3c16c: Deleting consensus metadata
I20260927 00:49:46.395584 15201 ts_tablet_manager.cc:1918] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:49:46.395776 15201 log.cc:1196] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/wal/wals/dc97813f83554e028981568de9f7c688
I20260927 00:49:46.396149 14691 catalog_manager.cc:4841] TS 03cbe42139b34d25a1f3f19c95c3c16c (127.0.98.3:42451): tablet dc97813f83554e028981568de9f7c688 (table test-table [id=dd2c86273d7240d08a596d424070e21f]) successfully deleted
I20260927 00:49:46.396248 15201 ts_tablet_manager.cc:1939] T dc97813f83554e028981568de9f7c688 P 06b92c2cf7e64899a2bfce641404ad87: Deleting consensus metadata
W20260927 00:49:46.396796 14692 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43081 (error 108)
I20260927 00:49:46.397928 14692 catalog_manager.cc:4841] TS 06b92c2cf7e64899a2bfce641404ad87 (127.0.98.2:35409): tablet dc97813f83554e028981568de9f7c688 (table test-table [id=dd2c86273d7240d08a596d424070e21f]) successfully deleted
W20260927 00:49:46.398314 15175 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:49:46.397801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43081 (local address 127.0.0.1:46248)
0927 00:49:46.397947 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:46.397964 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:46.398055 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:46.398204 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43081: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 00:49:46.398478 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 14899
W20260927 00:49:46.398578 14692 catalog_manager.cc:4584] TS e5bfcca01792442087b4108656ab072c (127.0.98.1:43081): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dc97813f83554e028981568de9f7c688: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43081: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 00:49:46.404181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 15036
I20260927 00:49:46.409379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 14674
I20260927 00:49:46.416438 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34607
--webserver_interface=127.0.98.62
--webserver_port=45967
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:49:46.532047 15203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:49:46.532495 15203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:49:46.532589 15203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:49:46.536895 15203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:49:46.536996 15203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:49:46.537036 15203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:49:46.537076 15203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:49:46.537112 15203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:49:46.542912 15203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:49:46.544134 15203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:49:46.545251 15203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:49:46.554365 15203 server_base.cc:1034] running on GCE node
W20260927 00:49:46.554349 15211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:46.554328 15209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:46.554630 15208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:46.555253 15203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:49:46.556452 15203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:49:46.557642 15203 hybrid_clock.cc:648] HybridClock initialized: now 1790470186557604 us; error 50 us; skew 500 ppm
Sep 27 00:49:46 dist-test-slave-86rs krb5kdc[14124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:49:46.560842 15203 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:49:46.561970 15203 webserver.cc:460] Webserver started at http://127.0.98.62:45967/ using document root <none> and password file <none>
I20260927 00:49:46.562680 15203 fs_manager.cc:359] Metadata directory not provided
I20260927 00:49:46.562778 15203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:49:46.567039 15203 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:49:46.569536 15218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:46.570861 15203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:49:46.571007 15203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "98ab283e0f91492083a657e1243d264b"
format_stamp: "Formatted at 2026-09-27 00:49:44 on dist-test-slave-86rs"
server_key: "dc0812c066da0be98532959f6ea74f55"
server_key_iv: "a9c0c0cce32783683fcd272a80019b63"
server_key_version: "encryptionkey@0"
I20260927 00:49:46.571583 15203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:49:46.590520 15203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:49:46.594094 15203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:49:46.594317 15203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:49:46.602255 15203 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34607
I20260927 00:49:46.602596 15270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34607 every 8 connection(s)
I20260927 00:49:46.603574 15203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:49:46.608192 15271 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:49:46.610033 15271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b: Bootstrap starting.
I20260927 00:49:46.613688 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 15203
I20260927 00:49:46.614159 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43081
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34607
--webserver_port=35185
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:49:46.628360 15271 log.cc:824] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b: Log is configured to *not* fsync() on all Append() calls
I20260927 00:49:46.640646 15271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:49:46.641088 15271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b: Bootstrap complete.
I20260927 00:49:46.645394 15271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ab283e0f91492083a657e1243d264b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34607 } }
I20260927 00:49:46.645856 15271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98ab283e0f91492083a657e1243d264b, State: Initialized, Role: FOLLOWER
I20260927 00:49:46.646438 15271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ab283e0f91492083a657e1243d264b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34607 } }
I20260927 00:49:46.646601 15271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:49:46.646687 15271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:49:46.646821 15271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [term 2 FOLLOWER]: Advancing to term 3
I20260927 00:49:46.648962 15271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ab283e0f91492083a657e1243d264b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34607 } }
I20260927 00:49:46.649315 15271 leader_election.cc:302] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98ab283e0f91492083a657e1243d264b; no voters:
I20260927 00:49:46.649701 15271 leader_election.cc:288] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 00:49:46.649802 15275 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [term 3 FOLLOWER]: Leader election won for term 3
I20260927 00:49:46.650008 15275 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [term 3 LEADER]: Becoming Leader. State: Replica: 98ab283e0f91492083a657e1243d264b, State: Running, Role: LEADER
I20260927 00:49:46.650310 15275 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ab283e0f91492083a657e1243d264b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34607 } }
I20260927 00:49:46.651016 15271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:49:46.651319 15275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98ab283e0f91492083a657e1243d264b. Latest consensus state: current_term: 3 leader_uuid: "98ab283e0f91492083a657e1243d264b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ab283e0f91492083a657e1243d264b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34607 } } }
I20260927 00:49:46.651441 15275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [sys.catalog]: This master's current role is: LEADER
I20260927 00:49:46.651815 15279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:49:46.651840 15276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "98ab283e0f91492083a657e1243d264b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ab283e0f91492083a657e1243d264b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34607 } } }
I20260927 00:49:46.652065 15276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ab283e0f91492083a657e1243d264b [sys.catalog]: This master's current role is: LEADER
I20260927 00:49:46.657640 15279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:49:46.658524 15279 catalog_manager.cc:1259] Loaded cluster ID: ca38794ed4a543a188b45b3426c8cee2
I20260927 00:49:46.658593 15279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 00:49:46.659499 15279 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4bf06549ad google::LogMessage::Fail() at ??:0
@ 0x7f4bf065788a google::LogMessage::SendToLog() at ??:0
@ 0x7f4bf06544de google::LogMessage::Flush() at ??:0
@ 0x7f4bf0655a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4bfaf93fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f4bfaf94d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4bfaf91579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f4bfafd5640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4bf28a63d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4bf28a6c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f4bf28a83f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4bf2897512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4bf3edb6db start_thread at ??:0
@ 0x7f4bef15261f clone at ??:0
W20260927 00:49:46.763913 15273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:49:46.764212 15273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:49:46.764328 15273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:49:46.768627 15273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:49:46.768695 15273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:49:46.768780 15273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:49:46.774478 15273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35185
--tserver_master_addrs=127.0.98.62:34607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:49:46.775698 15273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:49:46.776880 15273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:49:46.787766 15315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:46.787748 15312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:46.787806 15313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:46.788442 15273 server_base.cc:1034] running on GCE node
I20260927 00:49:46.788959 15273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:49:46.789921 15273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:49:46.791133 15273 hybrid_clock.cc:648] HybridClock initialized: now 1790470186791091 us; error 51 us; skew 500 ppm
Sep 27 00:49:46 dist-test-slave-86rs krb5kdc[14124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:49:46.794777 15273 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:49:46.795800 15273 webserver.cc:460] Webserver started at http://127.0.98.1:35185/ using document root <none> and password file <none>
I20260927 00:49:46.796518 15273 fs_manager.cc:359] Metadata directory not provided
I20260927 00:49:46.796614 15273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:49:46.801091 15273 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:49:46.803735 15322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:46.805356 15273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:49:46.805526 15273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "e5bfcca01792442087b4108656ab072c"
format_stamp: "Formatted at 2026-09-27 00:49:44 on dist-test-slave-86rs"
server_key: "67d708c3b58b8ffa4af2715c4812134f"
server_key_iv: "5eb5d584d11a602dec080be37068f7a2"
server_key_version: "encryptionkey@0"
I20260927 00:49:46.806155 15273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:49:46.838662 15273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:49:46.842293 15273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:49:46.842510 15273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:49:46.843148 15273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:49:46.844367 15329 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 00:49:46.850003 15273 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 00:49:46.850100 15273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 00:49:46.850209 15273 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 00:49:46.852053 15273 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 00:49:46.852124 15273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 00:49:46.852458 15329 tablet_bootstrap.cc:492] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c: Bootstrap starting.
I20260927 00:49:46.863108 15273 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43081
I20260927 00:49:46.863133 15436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43081 every 8 connection(s)
I20260927 00:49:46.864282 15273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:49:46.872044 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 15273
I20260927 00:49:46.872536 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35409
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34607
--webserver_port=33755
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:49:46.874886 15329 log.cc:824] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:49:46.882501 15329 tablet_bootstrap.cc:492] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:49:46.882936 15329 tablet_bootstrap.cc:492] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c: Bootstrap complete.
I20260927 00:49:46.884099 15329 ts_tablet_manager.cc:1397] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c: Time spent bootstrapping tablet: real 0.032s user 0.020s sys 0.007s
I20260927 00:49:46.886794 15329 raft_consensus.cc:348] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } } peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } }
I20260927 00:49:46.887332 15329 raft_consensus.cc:729] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5bfcca01792442087b4108656ab072c, State: Initialized, Role: FOLLOWER
I20260927 00:49:46.887881 15329 consensus_queue.cc:260] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } } peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } }
I20260927 00:49:46.888988 15329 ts_tablet_manager.cc:1428] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260927 00:49:46.910032 15439 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:49:46.866726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.1:54003)
0927 00:49:46.867244 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:46.867299 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:46.868164 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:46.908991 (+ 40827us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":170,"threads_started":1}
W20260927 00:49:46.910570 15437 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34607 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34607: Client connection negotiation failed: client connection to 127.0.98.62:34607: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 00:49:46.911340 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:46.911077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.1:47421)
0927 00:49:46.911207 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:46.911282 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 00:49:46.911988 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:46.911772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.1:41763)
0927 00:49:46.911879 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:46.911924 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 00:49:46.912142 15437 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:49:47.001504 15440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:49:47.001850 15440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:49:47.001935 15440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:49:47.006187 15440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:49:47.006309 15440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:49:47.006412 15440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:49:47.012487 15440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35409
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33755
--tserver_master_addrs=127.0.98.62:34607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:49:47.013695 15440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:49:47.014786 15440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:49:47.024870 15448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:47.024861 15450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:47.024861 15447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:47.025163 15440 server_base.cc:1034] running on GCE node
I20260927 00:49:47.025707 15440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:49:47.026357 15440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:49:47.027594 15440 hybrid_clock.cc:648] HybridClock initialized: now 1790470187027578 us; error 35 us; skew 500 ppm
Sep 27 00:49:47 dist-test-slave-86rs krb5kdc[14124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:49:47.030470 15440 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:49:47.031409 15440 webserver.cc:460] Webserver started at http://127.0.98.2:33755/ using document root <none> and password file <none>
I20260927 00:49:47.032027 15440 fs_manager.cc:359] Metadata directory not provided
I20260927 00:49:47.032126 15440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:49:47.036612 15440 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:49:47.038843 15457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:47.040325 15440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:49:47.040477 15440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "06b92c2cf7e64899a2bfce641404ad87"
format_stamp: "Formatted at 2026-09-27 00:49:44 on dist-test-slave-86rs"
server_key: "bf15d5a1b31ea7bddf7f3df1fdf6d7d2"
server_key_iv: "c7b397315648031aa244b245ca9eafed"
server_key_version: "encryptionkey@0"
I20260927 00:49:47.041078 15440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:49:47.075085 15440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:49:47.078657 15440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:49:47.078886 15440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:49:47.079542 15440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:49:47.080489 15440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:49:47.080561 15440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:47.080634 15440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:49:47.080683 15440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:47.090242 15440 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35409
I20260927 00:49:47.090262 15570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35409 every 8 connection(s)
I20260927 00:49:47.091356 15440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:49:47.095438 15573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:47.093697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.2:48459)
0927 00:49:47.094169 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:47.094279 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":113,"threads_started":1}
W20260927 00:49:47.095871 15571 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34607 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34607: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
I20260927 00:49:47.096498 15573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:47.096286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.2:50013)
0927 00:49:47.096388 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:47.096442 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 00:49:47.097034 15573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:47.096856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.2:45923)
0927 00:49:47.096944 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:47.096983 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 00:49:47.097214 15571 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 00:49:47.101205 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 15440
I20260927 00:49:47.101649 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42451
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34607
--webserver_port=36311
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:49:47.218622 15574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:49:47.218922 15574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:49:47.218967 15574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:49:47.223374 15574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:49:47.223448 15574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:49:47.223519 15574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:49:47.229279 15574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36311
--tserver_master_addrs=127.0.98.62:34607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:49:47.230400 15574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:49:47.231462 15574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:49:47.241851 15579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:47.241851 15580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:49:47.241919 15582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:49:47.242084 15574 server_base.cc:1034] running on GCE node
I20260927 00:49:47.242646 15574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:49:47.243337 15574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:49:47.244518 15574 hybrid_clock.cc:648] HybridClock initialized: now 1790470187244501 us; error 32 us; skew 500 ppm
Sep 27 00:49:47 dist-test-slave-86rs krb5kdc[14124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:49:47.247424 15574 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:49:47.248530 15574 webserver.cc:460] Webserver started at http://127.0.98.3:36311/ using document root <none> and password file <none>
I20260927 00:49:47.249120 15574 fs_manager.cc:359] Metadata directory not provided
I20260927 00:49:47.249212 15574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:49:47.253230 15574 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:49:47.255471 15589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:47.256893 15574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:49:47.257035 15574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "03cbe42139b34d25a1f3f19c95c3c16c"
format_stamp: "Formatted at 2026-09-27 00:49:44 on dist-test-slave-86rs"
server_key: "10e9c1454c1926fd4ce32a7d12111343"
server_key_iv: "f061c013a5a5c0ca131e23d7ae533502"
server_key_version: "encryptionkey@0"
I20260927 00:49:47.257616 15574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:49:47.276041 15574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:49:47.279392 15574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:49:47.279624 15574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:49:47.280337 15574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:49:47.281214 15574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:49:47.281261 15574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:47.281302 15574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:49:47.281318 15574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:49:47.290166 15574 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42451
I20260927 00:49:47.290241 15702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42451 every 8 connection(s)
I20260927 00:49:47.291301 15574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:49:47.295547 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:47.293689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.3:57751)
0927 00:49:47.294205 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:47.294337 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":122,"threads_started":1}
W20260927 00:49:47.296099 15703 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34607 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34607: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
I20260927 00:49:47.296836 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:47.296583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.3:45277)
0927 00:49:47.296722 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:47.296776 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 00:49:47.297528 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:47.297284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.3:54899)
0927 00:49:47.297396 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:47.297441 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 00:49:47.297693 15703 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 00:49:47.298754 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 15574
I20260927 00:49:47.913096 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:47.912676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.1:49295)
0927 00:49:47.912923 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:47.912980 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":75,"threads_started":1}
I20260927 00:49:48.098322 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:48.097877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.2:46293)
0927 00:49:48.098156 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:48.098216 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":103,"threads_started":1}
I20260927 00:49:48.196308 15709 raft_consensus.cc:482] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:49:48.196576 15709 raft_consensus.cc:504] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } } peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } }
I20260927 00:49:48.197834 15709 leader_election.cc:288] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03cbe42139b34d25a1f3f19c95c3c16c (127.0.98.3:42451), 06b92c2cf7e64899a2bfce641404ad87 (127.0.98.2:35409)
Sep 27 00:49:48 dist-test-slave-86rs krb5kdc[14124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 27 00:49:48 dist-test-slave-86rs krb5kdc[14124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260927 00:49:48.213186 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:48.198390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54871 (local address 127.0.98.3:42451)
0927 00:49:48.198825 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:48.198838 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:48.198880 (+ 42us) server_negotiation.cc:407] Connection header received
0927 00:49:48.199665 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:48.199673 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:48.200105 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:48.200490 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:48.201771 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:48.202749 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:48.203981 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:48.204535 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:48.204722 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:48.207938 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:48.207947 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:48.207964 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:48.208014 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:48.209977 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:48.210753 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:48.210761 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:48.210767 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:48.210900 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:48.211375 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:48.211378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:48.211380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:48.211835 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:48.211936 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:48.212750 (+ 814us) server_negotiation.cc:299] Negotiation successful
0927 00:49:48.213011 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":88,"threads_started":1}
I20260927 00:49:48.213172 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:48.198130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35409 (local address 127.0.98.1:45773)
0927 00:49:48.198315 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:48.198333 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:48.198463 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:48.200548 (+ 2085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:48.200555 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:48.201107 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:49:48.201590 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:48.201606 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:48.202916 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:48.202925 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:48.203839 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:48.203848 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:48.204669 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:48.204673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:48.204910 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:48.205615 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:49:48.205631 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:49:48.207358 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:49:48.210114 (+ 2756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:48.210120 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:48.210137 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:48.210609 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:48.211085 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:48.211088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:48.211090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:48.211285 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:48.212136 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:49:48.212139 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:49:48.212399 (+ 260us) client_negotiation.cc:769] Sending connection context
0927 00:49:48.212598 (+ 199us) client_negotiation.cc:240] Negotiation successful
0927 00:49:48.212969 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":122,"spinlock_wait_cycles":26624}
I20260927 00:49:48.213457 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:48.198130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42451 (local address 127.0.98.1:54871)
0927 00:49:48.198721 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:48.198736 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:49:48.198841 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:49:48.200548 (+ 1707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:49:48.200555 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:49:48.201108 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:49:48.201590 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:48.201606 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:48.202915 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:48.202919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:48.203841 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:49:48.203851 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:49:48.204669 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:49:48.204673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:49:48.204909 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:48.205666 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:49:48.205679 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:49:48.207706 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:49:48.210138 (+ 2432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:48.210145 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:48.210148 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:48.210622 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:48.211085 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:49:48.211088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:49:48.211090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:49:48.211273 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:49:48.212077 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:49:48.212083 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:49:48.212399 (+ 316us) client_negotiation.cc:769] Sending connection context
0927 00:49:48.212597 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 00:49:48.213056 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":537,"spinlock_wait_cycles":26496,"thread_start_us":99,"threads_started":1}
I20260927 00:49:48.213243 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:48.198405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45773 (local address 127.0.98.2:35409)
0927 00:49:48.198747 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0927 00:49:48.198771 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 00:49:48.198810 (+ 39us) server_negotiation.cc:407] Connection header received
0927 00:49:48.199648 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:49:48.199660 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:49:48.200106 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:49:48.200489 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:49:48.201777 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:48.202748 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:48.203976 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:49:48.204537 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:49:48.204720 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:49:48.207618 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:49:48.207623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:49:48.207636 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:49:48.207665 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:49:48.209977 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:48.210785 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:48.210793 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:48.210798 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:48.210959 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:49:48.211420 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:49:48.211423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:49:48.211424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:49:48.211834 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:49:48.212116 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 00:49:48.212732 (+ 616us) server_negotiation.cc:299] Negotiation successful
0927 00:49:48.213043 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":70,"threads_started":1}
W20260927 00:49:48.215896 15323 leader_election.cc:341] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 03cbe42139b34d25a1f3f19c95c3c16c (127.0.98.3:42451): Not found: Tablet not found: dc97813f83554e028981568de9f7c688
W20260927 00:49:48.216121 15324 leader_election.cc:341] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 06b92c2cf7e64899a2bfce641404ad87 (127.0.98.2:35409): Not found: Tablet not found: dc97813f83554e028981568de9f7c688
I20260927 00:49:48.216225 15324 leader_election.cc:302] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bfcca01792442087b4108656ab072c; no voters: 03cbe42139b34d25a1f3f19c95c3c16c, 06b92c2cf7e64899a2bfce641404ad87
I20260927 00:49:48.216601 15709 raft_consensus.cc:2726] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:49:48.298847 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:48.298328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.3:60553)
0927 00:49:48.298676 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:48.298741 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":86,"threads_started":1}
I20260927 00:49:48.914326 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:48.913892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.1:33509)
0927 00:49:48.914159 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:48.914224 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":74,"threads_started":1}
I20260927 00:49:49.099601 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:49.099213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.2:59553)
0927 00:49:49.099451 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:49.099504 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":72,"threads_started":1}
I20260927 00:49:49.300098 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:49.299655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.3:56813)
0927 00:49:49.299919 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:49.299983 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":74,"threads_started":1}
I20260927 00:49:49.792593 15717 raft_consensus.cc:482] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:49:49.792753 15717 raft_consensus.cc:504] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } } peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } }
I20260927 00:49:49.793255 15717 leader_election.cc:288] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03cbe42139b34d25a1f3f19c95c3c16c (127.0.98.3:42451), 06b92c2cf7e64899a2bfce641404ad87 (127.0.98.2:35409)
W20260927 00:49:49.794337 15324 leader_election.cc:341] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 06b92c2cf7e64899a2bfce641404ad87 (127.0.98.2:35409): Not found: Tablet not found: dc97813f83554e028981568de9f7c688
W20260927 00:49:49.794435 15323 leader_election.cc:341] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 03cbe42139b34d25a1f3f19c95c3c16c (127.0.98.3:42451): Not found: Tablet not found: dc97813f83554e028981568de9f7c688
I20260927 00:49:49.794541 15323 leader_election.cc:302] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bfcca01792442087b4108656ab072c; no voters: 03cbe42139b34d25a1f3f19c95c3c16c, 06b92c2cf7e64899a2bfce641404ad87
I20260927 00:49:49.794710 15717 raft_consensus.cc:2726] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:49:49.915838 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:49.915311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.1:44901)
0927 00:49:49.915643 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:49.915708 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":116,"threads_started":1}
I20260927 00:49:50.101204 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:50.100595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.2:59567)
0927 00:49:50.100988 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:50.101058 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":155,"threads_started":1}
I20260927 00:49:50.301662 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:50.301086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.3:47407)
0927 00:49:50.301453 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:50.301526 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":126,"threads_started":1}
I20260927 00:49:50.917251 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:50.916772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.1:35629)
0927 00:49:50.917071 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:50.917136 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":79,"threads_started":1}
I20260927 00:49:51.102759 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:51.102251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.2:52999)
0927 00:49:51.102573 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:51.102641 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":144,"threads_started":1}
I20260927 00:49:51.303139 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:51.302639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.3:56247)
0927 00:49:51.302954 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:51.303017 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":86,"threads_started":1}
I20260927 00:49:51.918764 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:51.918271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.1:50167)
0927 00:49:51.918578 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:51.918661 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":74,"threads_started":1}
I20260927 00:49:52.104368 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:52.103805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.2:42857)
0927 00:49:52.104116 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:52.104181 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":87,"threads_started":1}
I20260927 00:49:52.304690 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:52.304082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.3:50329)
0927 00:49:52.304450 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:52.304532 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":168,"threads_started":1}
I20260927 00:49:52.920096 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:52.919632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.1:48681)
0927 00:49:52.919935 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:52.919995 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":93,"threads_started":1}
I20260927 00:49:53.105546 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:53.105121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.2:42861)
0927 00:49:53.105389 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:53.105445 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":87,"threads_started":1}
I20260927 00:49:53.117198 15729 raft_consensus.cc:482] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:49:53.117347 15729 raft_consensus.cc:504] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } } peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } }
I20260927 00:49:53.117959 15729 leader_election.cc:288] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03cbe42139b34d25a1f3f19c95c3c16c (127.0.98.3:42451), 06b92c2cf7e64899a2bfce641404ad87 (127.0.98.2:35409)
W20260927 00:49:53.118808 15323 leader_election.cc:341] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 03cbe42139b34d25a1f3f19c95c3c16c (127.0.98.3:42451): Not found: Tablet not found: dc97813f83554e028981568de9f7c688
W20260927 00:49:53.118901 15324 leader_election.cc:341] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 06b92c2cf7e64899a2bfce641404ad87 (127.0.98.2:35409): Not found: Tablet not found: dc97813f83554e028981568de9f7c688
I20260927 00:49:53.119022 15324 leader_election.cc:302] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bfcca01792442087b4108656ab072c; no voters: 03cbe42139b34d25a1f3f19c95c3c16c, 06b92c2cf7e64899a2bfce641404ad87
I20260927 00:49:53.119230 15729 raft_consensus.cc:2726] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:49:53.305972 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:53.305547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.3:50779)
0927 00:49:53.305813 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:53.305870 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":79,"threads_started":1}
I20260927 00:49:53.921434 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:53.920980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.1:52695)
0927 00:49:53.921272 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:53.921335 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":96,"threads_started":1}
I20260927 00:49:54.106907 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:54.106372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.2:44019)
0927 00:49:54.106745 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:54.106806 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":87,"threads_started":1}
I20260927 00:49:54.307262 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:54.306766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.3:52315)
0927 00:49:54.307103 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:54.307161 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":72,"threads_started":1}
I20260927 00:49:54.922864 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:54.922378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.1:46275)
0927 00:49:54.922685 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:54.922748 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":88,"threads_started":1}
I20260927 00:49:55.108361 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:55.107844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.2:33111)
0927 00:49:55.108135 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:55.108199 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":95,"threads_started":1}
I20260927 00:49:55.308847 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:55.308354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.3:58653)
0927 00:49:55.308671 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:55.308733 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":76,"threads_started":1}
I20260927 00:49:55.924363 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:55.923772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.1:47545)
0927 00:49:55.924108 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:55.924176 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":94,"threads_started":1}
I20260927 00:49:56.109829 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:56.109297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.2:59267)
0927 00:49:56.109652 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:56.109721 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":110,"threads_started":1}
I20260927 00:49:56.310246 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:56.309725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.3:57223)
0927 00:49:56.310061 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:56.310142 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":85,"threads_started":1}
I20260927 00:49:56.802368 15740 raft_consensus.cc:482] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:49:56.802506 15740 raft_consensus.cc:504] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } } peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } }
I20260927 00:49:56.803050 15740 leader_election.cc:288] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03cbe42139b34d25a1f3f19c95c3c16c (127.0.98.3:42451), 06b92c2cf7e64899a2bfce641404ad87 (127.0.98.2:35409)
W20260927 00:49:56.803885 15323 leader_election.cc:341] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 03cbe42139b34d25a1f3f19c95c3c16c (127.0.98.3:42451): Not found: Tablet not found: dc97813f83554e028981568de9f7c688
W20260927 00:49:56.804000 15324 leader_election.cc:341] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 06b92c2cf7e64899a2bfce641404ad87 (127.0.98.2:35409): Not found: Tablet not found: dc97813f83554e028981568de9f7c688
I20260927 00:49:56.804131 15324 leader_election.cc:302] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bfcca01792442087b4108656ab072c; no voters: 03cbe42139b34d25a1f3f19c95c3c16c, 06b92c2cf7e64899a2bfce641404ad87
I20260927 00:49:56.804316 15740 raft_consensus.cc:2726] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:49:56.925784 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:56.925318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.1:38385)
0927 00:49:56.925620 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:56.925683 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":84,"threads_started":1}
I20260927 00:49:57.111193 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:57.110724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.2:58661)
0927 00:49:57.111026 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:57.111090 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":91,"threads_started":1}
I20260927 00:49:57.311568 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:57.311131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.3:36419)
0927 00:49:57.311400 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:57.311464 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":84,"threads_started":1}
I20260927 00:49:57.927207 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:57.926736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.1:49783)
0927 00:49:57.927044 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:57.927107 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":82,"threads_started":1}
I20260927 00:49:58.112752 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:58.112182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.2:56601)
0927 00:49:58.112581 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:58.112648 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":160,"threads_started":1}
I20260927 00:49:58.313272 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:58.312801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.3:52561)
0927 00:49:58.313106 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:58.313170 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":87,"threads_started":1}
I20260927 00:49:58.928715 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:58.928034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.1:40457)
0927 00:49:58.928497 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:58.928580 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":98,"threads_started":1}
I20260927 00:49:59.114009 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:59.113553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.2:56901)
0927 00:49:59.113848 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:59.113908 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":89,"threads_started":1}
I20260927 00:49:59.223682 15749 raft_consensus.cc:482] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:49:59.223816 15749 raft_consensus.cc:504] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03cbe42139b34d25a1f3f19c95c3c16c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42451 } } peers { permanent_uuid: "06b92c2cf7e64899a2bfce641404ad87" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "e5bfcca01792442087b4108656ab072c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43081 } }
I20260927 00:49:59.224354 15749 leader_election.cc:288] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03cbe42139b34d25a1f3f19c95c3c16c (127.0.98.3:42451), 06b92c2cf7e64899a2bfce641404ad87 (127.0.98.2:35409)
W20260927 00:49:59.225237 15323 leader_election.cc:341] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 03cbe42139b34d25a1f3f19c95c3c16c (127.0.98.3:42451): Not found: Tablet not found: dc97813f83554e028981568de9f7c688
W20260927 00:49:59.225330 15324 leader_election.cc:341] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 06b92c2cf7e64899a2bfce641404ad87 (127.0.98.2:35409): Not found: Tablet not found: dc97813f83554e028981568de9f7c688
I20260927 00:49:59.225417 15324 leader_election.cc:302] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5bfcca01792442087b4108656ab072c; no voters: 03cbe42139b34d25a1f3f19c95c3c16c, 06b92c2cf7e64899a2bfce641404ad87
I20260927 00:49:59.225549 15749 raft_consensus.cc:2726] T dc97813f83554e028981568de9f7c688 P e5bfcca01792442087b4108656ab072c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:49:59.314604 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:59.314118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.3:44175)
0927 00:49:59.314441 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:59.314504 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":86,"threads_started":1}
I20260927 00:49:59.929989 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:49:59.929556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.1:33113)
0927 00:49:59.929833 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 00:49:59.929888 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":91,"threads_started":1}
I20260927 00:50:00.115365 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:00.114892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.2:53713)
0927 00:50:00.115194 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:00.115261 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":86,"threads_started":1}
I20260927 00:50:00.316329 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:00.315772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.3:37007)
0927 00:50:00.316095 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:00.316153 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":91,"threads_started":1}
I20260927 00:50:00.931391 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:00.930939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.1:42179)
0927 00:50:00.931227 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:00.931288 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":99,"threads_started":1}
I20260927 00:50:01.116791 15755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:01.116328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.2:49159)
0927 00:50:01.116628 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:01.116689 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":102,"threads_started":1}
I20260927 00:50:01.317669 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:01.317145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.3:42957)
0927 00:50:01.317487 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:01.317553 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":82,"threads_started":1}
I20260927 00:50:01.933205 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:01.932670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.1:51681)
0927 00:50:01.933017 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:01.933078 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":125,"threads_started":1}
I20260927 00:50:02.118474 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:02.117914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34607 (local address 127.0.98.2:43851)
0927 00:50:02.118264 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:02.118342 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":98,"threads_started":1}
I20260927 00:50:02.302884 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 15273
I20260927 00:50:02.310037 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 15440
I20260927 00:50:02.316212 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 15574
2026-09-27T00:50:02Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19600 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:50:02 dist-test-slave-86rs krb5kdc[15761](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:50:02 dist-test-slave-86rs krb5kdc[15761](info): set up 2 sockets
Sep 27 00:50:02 dist-test-slave-86rs krb5kdc[15761](info): commencing operation
krb5kdc: starting...
W20260927 00:50:03.372890 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:50:03 dist-test-slave-86rs krb5kdc[15761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:50:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:50:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:50:03.515774 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34319
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:50:03.643441 15774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:50:03.643850 15774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:50:03.643947 15774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:50:03.648329 15774 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 00:50:03.648437 15774 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 00:50:03.648461 15774 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 00:50:03.648487 15774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:50:03.648532 15774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:50:03.648597 15774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:50:03.648653 15774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:50:03.648689 15774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:50:03.654707 15774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:50:03.656209 15774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:50:03.657692 15774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:50:03.668514 15779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:50:03.668538 15780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:50:03.668609 15774 server_base.cc:1034] running on GCE node
W20260927 00:50:03.668516 15782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:50:03.669612 15774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:50:03.671065 15774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:50:03.672297 15774 hybrid_clock.cc:648] HybridClock initialized: now 1790470203672257 us; error 53 us; skew 500 ppm
Sep 27 00:50:03 dist-test-slave-86rs krb5kdc[15761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470203, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:50:03.675345 15774 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:50:03.676502 15774 webserver.cc:460] Webserver started at http://127.0.98.62:41639/ using document root <none> and password file <none>
I20260927 00:50:03.677188 15774 fs_manager.cc:359] Metadata directory not provided
I20260927 00:50:03.677284 15774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:50:03.677567 15774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:50:03.680074 15774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "f491678175874e80b10a2e572a51ccd1"
format_stamp: "Formatted at 2026-09-27 00:50:03 on dist-test-slave-86rs"
server_key: "76045a0732c9276322172d0aa3f8cc2e"
server_key_iv: "b8cc60c006346ca8afe1a7ff8ac77b7b"
server_key_version: "encryptionkey@0"
I20260927 00:50:03.680835 15774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "f491678175874e80b10a2e572a51ccd1"
format_stamp: "Formatted at 2026-09-27 00:50:03 on dist-test-slave-86rs"
server_key: "76045a0732c9276322172d0aa3f8cc2e"
server_key_iv: "b8cc60c006346ca8afe1a7ff8ac77b7b"
server_key_version: "encryptionkey@0"
I20260927 00:50:03.685379 15774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:50:03.688362 15789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:50:03.689875 15774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:50:03.690037 15774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "f491678175874e80b10a2e572a51ccd1"
format_stamp: "Formatted at 2026-09-27 00:50:03 on dist-test-slave-86rs"
server_key: "76045a0732c9276322172d0aa3f8cc2e"
server_key_iv: "b8cc60c006346ca8afe1a7ff8ac77b7b"
server_key_version: "encryptionkey@0"
I20260927 00:50:03.690178 15774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:50:03.705035 15774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:50:03.708643 15774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:50:03.708881 15774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:50:03.717360 15774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34319
I20260927 00:50:03.717689 15841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34319 every 8 connection(s)
I20260927 00:50:03.718669 15774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:50:03.722333 15842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:50:03.723462 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 15774
I20260927 00:50:03.723598 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:50:03.723995 392 external_mini_cluster.cc:1428] Setting key 5c2e702d18e30d49083d072089d2e604
I20260927 00:50:03.729975 15842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1: Bootstrap starting.
I20260927 00:50:03.732925 15842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1: Neither blocks nor log segments found. Creating new log.
I20260927 00:50:03.733855 15842 log.cc:824] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:50:03 dist-test-slave-86rs krb5kdc[15761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:50:03.736964 15842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1: No bootstrap required, opened a new log
I20260927 00:50:03.741020 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:03.725833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43000 (local address 127.0.98.62:34319)
0927 00:50:03.726398 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0927 00:50:03.726417 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:50:03.726461 (+ 44us) server_negotiation.cc:407] Connection header received
0927 00:50:03.727212 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:50:03.727236 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:50:03.727677 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:50:03.728102 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:50:03.729002 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:03.730248 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:03.731166 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:03.731710 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:03.731895 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:03.734958 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:50:03.734967 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:50:03.734992 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:50:03.735024 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:50:03.737246 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:50:03.737731 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:50:03.737738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:50:03.737746 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:50:03.737897 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:50:03.738162 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:50:03.738165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:50:03.738167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:50:03.738745 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:50:03.738958 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 00:50:03.739236 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 00:50:03.739632 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":163,"threads_started":1}
I20260927 00:50:03.742460 15842 raft_consensus.cc:348] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f491678175874e80b10a2e572a51ccd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } }
I20260927 00:50:03.742794 15842 raft_consensus.cc:374] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:50:03.742877 15842 raft_consensus.cc:729] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f491678175874e80b10a2e572a51ccd1, State: Initialized, Role: FOLLOWER
I20260927 00:50:03.743391 15842 consensus_queue.cc:260] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f491678175874e80b10a2e572a51ccd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } }
I20260927 00:50:03.743546 15842 raft_consensus.cc:388] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:50:03.743611 15842 raft_consensus.cc:482] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:50:03.743717 15842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:50:03.744988 15842 raft_consensus.cc:504] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f491678175874e80b10a2e572a51ccd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } }
I20260927 00:50:03.745417 15842 leader_election.cc:302] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f491678175874e80b10a2e572a51ccd1; no voters:
I20260927 00:50:03.745864 15842 leader_election.cc:288] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:50:03.746021 15847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:50:03.746598 15847 raft_consensus.cc:686] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [term 1 LEADER]: Becoming Leader. State: Replica: f491678175874e80b10a2e572a51ccd1, State: Running, Role: LEADER
I20260927 00:50:03.747041 15847 consensus_queue.cc:237] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f491678175874e80b10a2e572a51ccd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } }
I20260927 00:50:03.747965 15842 sys_catalog.cc:564] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:50:03.749423 15848 sys_catalog.cc:455] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f491678175874e80b10a2e572a51ccd1. Latest consensus state: current_term: 1 leader_uuid: "f491678175874e80b10a2e572a51ccd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f491678175874e80b10a2e572a51ccd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } } }
I20260927 00:50:03.749462 15849 sys_catalog.cc:455] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f491678175874e80b10a2e572a51ccd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f491678175874e80b10a2e572a51ccd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } } }
I20260927 00:50:03.749577 15848 sys_catalog.cc:458] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [sys.catalog]: This master's current role is: LEADER
I20260927 00:50:03.749577 15849 sys_catalog.cc:458] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [sys.catalog]: This master's current role is: LEADER
I20260927 00:50:03.750058 15856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:50:03.754484 15856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:50:03.762528 15856 catalog_manager.cc:1347] Generated new cluster ID: f18e887dadde4191a8a38af6fa42730e
I20260927 00:50:03.762652 15856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:50:03.769273 15856 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:50:03.770467 15856 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:50:03.776433 15856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1: Generated new TSK 0
I20260927 00:50:03.777469 15856 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:50:03.821632 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34319
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:50:03.997723 15868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:50:03.998135 15868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:50:03.998191 15868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:50:04.002918 15868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:50:04.003023 15868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:50:04.003098 15868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:50:04.009685 15868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:50:04.011302 15868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:50:04.012898 15868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:50:04.025846 15873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:50:04.025836 15874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:50:04.025832 15876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:50:04.026168 15868 server_base.cc:1034] running on GCE node
I20260927 00:50:04.026762 15868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:50:04.027659 15868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:50:04.028899 15868 hybrid_clock.cc:648] HybridClock initialized: now 1790470204028872 us; error 40 us; skew 500 ppm
Sep 27 00:50:04 dist-test-slave-86rs krb5kdc[15761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:50:04.032627 15868 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:50:04.033902 15868 webserver.cc:460] Webserver started at http://127.0.98.1:45759/ using document root <none> and password file <none>
I20260927 00:50:04.034703 15868 fs_manager.cc:359] Metadata directory not provided
I20260927 00:50:04.034817 15868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:50:04.035122 15868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:50:04.038130 15868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/data/instance:
uuid: "a52f4e455782487687acb63ecff91e1a"
format_stamp: "Formatted at 2026-09-27 00:50:04 on dist-test-slave-86rs"
server_key: "a6c3acbb676d08ca37fbc3b5e6ae7f57"
server_key_iv: "4303f9cd97d5b3901cac3b9b76c016cd"
server_key_version: "encryptionkey@0"
I20260927 00:50:04.039232 15868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a52f4e455782487687acb63ecff91e1a"
format_stamp: "Formatted at 2026-09-27 00:50:04 on dist-test-slave-86rs"
server_key: "a6c3acbb676d08ca37fbc3b5e6ae7f57"
server_key_iv: "4303f9cd97d5b3901cac3b9b76c016cd"
server_key_version: "encryptionkey@0"
I20260927 00:50:04.045571 15868 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 00:50:04.049391 15883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:50:04.051167 15868 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260927 00:50:04.051378 15868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "a52f4e455782487687acb63ecff91e1a"
format_stamp: "Formatted at 2026-09-27 00:50:04 on dist-test-slave-86rs"
server_key: "a6c3acbb676d08ca37fbc3b5e6ae7f57"
server_key_iv: "4303f9cd97d5b3901cac3b9b76c016cd"
server_key_version: "encryptionkey@0"
I20260927 00:50:04.051544 15868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:50:04.065703 15868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:50:04.070231 15868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:50:04.070533 15868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:50:04.071413 15868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:50:04.072777 15868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:50:04.072856 15868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:50:04.072968 15868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:50:04.073015 15868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:50:04.084932 15868 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40125
I20260927 00:50:04.084993 15996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40125 every 8 connection(s)
I20260927 00:50:04.086231 15868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:50:04.093403 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 15868
I20260927 00:50:04.093564 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:50:04.093936 392 external_mini_cluster.cc:1428] Setting key 8ce986914d4722e01dd1e99fcc84557d
Sep 27 00:50:04 dist-test-slave-86rs krb5kdc[15761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:50:04.106874 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:04.089059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34175 (local address 127.0.98.62:34319)
0927 00:50:04.089278 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 00:50:04.089283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:50:04.090482 (+ 1199us) server_negotiation.cc:407] Connection header received
0927 00:50:04.092089 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:50:04.092093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:50:04.092165 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:50:04.092349 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:50:04.094533 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.095504 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.097565 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.097880 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.097982 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:04.101446 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:50:04.101453 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:50:04.101460 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:50:04.101502 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:50:04.103778 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:50:04.104616 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:50:04.104622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:50:04.104625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:50:04.104755 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:50:04.105191 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:50:04.105196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:50:04.105199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:50:04.105522 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:50:04.105650 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 00:50:04.106418 (+ 768us) server_negotiation.cc:299] Negotiation successful
0927 00:50:04.106617 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 00:50:04.108081 15999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:04.089637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.1:34175)
0927 00:50:04.090284 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:04.090341 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 00:50:04.091797 (+ 1456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:50:04.092652 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:50:04.092663 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:50:04.093317 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:50:04.094251 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:04.094283 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.095681 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.095684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:04.097409 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:04.097426 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.098076 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.098083 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:04.098370 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:04.099135 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:50:04.099155 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:50:04.101177 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:50:04.103970 (+ 2793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:50:04.103978 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:50:04.104000 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:50:04.104480 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:50:04.104877 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:50:04.104880 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:50:04.104881 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:50:04.105062 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:50:04.105714 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:50:04.105723 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:50:04.106094 (+ 371us) client_negotiation.cc:769] Sending connection context
0927 00:50:04.106307 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 00:50:04.106775 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":188,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:50:04.110303 15997 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34319
I20260927 00:50:04.110700 15997 heartbeater.cc:464] Registering TS with master...
I20260927 00:50:04.111497 15997 heartbeater.cc:511] Master 127.0.98.62:34319 requested a full tablet report, sending...
I20260927 00:50:04.114023 15806 ts_manager.cc:194] Registered new tserver with Master: a52f4e455782487687acb63ecff91e1a (127.0.98.1:40125)
I20260927 00:50:04.116104 15806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:50:04.131919 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34319
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:50:04.276093 16002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:50:04.276590 16002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:50:04.276700 16002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:50:04.281142 16002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:50:04.281219 16002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:50:04.281286 16002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:50:04.287226 16002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:50:04.288842 16002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:50:04.290194 16002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:50:04.302284 16008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:50:04.302269 16007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:50:04.302341 16002 server_base.cc:1034] running on GCE node
W20260927 00:50:04.302289 16010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:50:04.303001 16002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:50:04.303855 16002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:50:04.305025 16002 hybrid_clock.cc:648] HybridClock initialized: now 1790470204305003 us; error 35 us; skew 500 ppm
Sep 27 00:50:04 dist-test-slave-86rs krb5kdc[15761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:50:04.308197 16002 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:50:04.309314 16002 webserver.cc:460] Webserver started at http://127.0.98.2:34355/ using document root <none> and password file <none>
I20260927 00:50:04.310042 16002 fs_manager.cc:359] Metadata directory not provided
I20260927 00:50:04.310109 16002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:50:04.310314 16002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:50:04.312804 16002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/data/instance:
uuid: "5dd062a9695e41c59a75ef425fb7deed"
format_stamp: "Formatted at 2026-09-27 00:50:04 on dist-test-slave-86rs"
server_key: "e54e64010f47e010a8da134e7fe16a4c"
server_key_iv: "67e542ec782de4bf0e996c80619b0aa5"
server_key_version: "encryptionkey@0"
I20260927 00:50:04.313398 16002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5dd062a9695e41c59a75ef425fb7deed"
format_stamp: "Formatted at 2026-09-27 00:50:04 on dist-test-slave-86rs"
server_key: "e54e64010f47e010a8da134e7fe16a4c"
server_key_iv: "67e542ec782de4bf0e996c80619b0aa5"
server_key_version: "encryptionkey@0"
I20260927 00:50:04.317775 16002 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:50:04.320706 16017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:50:04.322270 16002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:50:04.322376 16002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "5dd062a9695e41c59a75ef425fb7deed"
format_stamp: "Formatted at 2026-09-27 00:50:04 on dist-test-slave-86rs"
server_key: "e54e64010f47e010a8da134e7fe16a4c"
server_key_iv: "67e542ec782de4bf0e996c80619b0aa5"
server_key_version: "encryptionkey@0"
I20260927 00:50:04.322476 16002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:50:04.371016 16002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:50:04.374529 16002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:50:04.374720 16002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:50:04.375418 16002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:50:04.376457 16002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:50:04.376505 16002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:50:04.376550 16002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:50:04.376566 16002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:50:04.387080 16002 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43741
I20260927 00:50:04.387104 16130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43741 every 8 connection(s)
I20260927 00:50:04.388306 16002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:50:04.391993 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 16002
I20260927 00:50:04.392163 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:50:04.392565 392 external_mini_cluster.cc:1428] Setting key cf644e2b256dca3a82f0396455cb4066
Sep 27 00:50:04 dist-test-slave-86rs krb5kdc[15761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:50:04.407081 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:04.390842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38579 (local address 127.0.98.62:34319)
0927 00:50:04.391029 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 00:50:04.391033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:50:04.392529 (+ 1496us) server_negotiation.cc:407] Connection header received
0927 00:50:04.394454 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:50:04.394457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:50:04.394524 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:50:04.394619 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:50:04.396281 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.397043 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.399058 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.399325 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.399415 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:04.403052 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:50:04.403055 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:50:04.403058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:50:04.403089 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:50:04.404780 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:50:04.405337 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:50:04.405341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:50:04.405343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:50:04.405439 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:50:04.405797 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:50:04.405802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:50:04.405804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:50:04.406020 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:50:04.406130 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:50:04.406729 (+ 599us) server_negotiation.cc:299] Negotiation successful
0927 00:50:04.406883 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:50:04.408618 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:04.391420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.2:38579)
0927 00:50:04.392277 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:04.392364 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0927 00:50:04.394114 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:50:04.394859 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:50:04.394870 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:50:04.395389 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:50:04.396070 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:04.396087 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.397218 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.397225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:04.398914 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:04.398931 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.399513 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.399520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:04.399805 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:04.400688 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:50:04.400719 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:50:04.402844 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:50:04.404908 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:50:04.404915 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:50:04.404936 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:50:04.405228 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:50:04.405538 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:50:04.405541 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:50:04.405544 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:50:04.405701 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:50:04.406127 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:50:04.406136 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:50:04.406457 (+ 321us) client_negotiation.cc:769] Sending connection context
0927 00:50:04.406683 (+ 226us) client_negotiation.cc:240] Negotiation successful
0927 00:50:04.407094 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":255,"threads_started":1}
I20260927 00:50:04.410190 16131 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34319
I20260927 00:50:04.410506 16131 heartbeater.cc:464] Registering TS with master...
I20260927 00:50:04.411145 16131 heartbeater.cc:511] Master 127.0.98.62:34319 requested a full tablet report, sending...
I20260927 00:50:04.412593 15806 ts_manager.cc:194] Registered new tserver with Master: 5dd062a9695e41c59a75ef425fb7deed (127.0.98.2:43741)
I20260927 00:50:04.413617 15806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:50:04.427762 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34319
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:50:04.562527 16136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:50:04.562850 16136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:50:04.562928 16136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:50:04.567071 16136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:50:04.567137 16136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:50:04.567240 16136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:50:04.572919 16136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:50:04.574481 16136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:50:04.575837 16136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:50:04.587723 16144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:50:04.587740 16142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:50:04.587755 16141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:50:04.588131 16136 server_base.cc:1034] running on GCE node
I20260927 00:50:04.588686 16136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:50:04.589519 16136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:50:04.590752 16136 hybrid_clock.cc:648] HybridClock initialized: now 1790470204590710 us; error 57 us; skew 500 ppm
Sep 27 00:50:04 dist-test-slave-86rs krb5kdc[15761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:50:04.594460 16136 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:50:04.595644 16136 webserver.cc:460] Webserver started at http://127.0.98.3:38461/ using document root <none> and password file <none>
I20260927 00:50:04.596426 16136 fs_manager.cc:359] Metadata directory not provided
I20260927 00:50:04.596500 16136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:50:04.596716 16136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:50:04.599284 16136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/data/instance:
uuid: "ba6f939d15304d43b4212ea876b03187"
format_stamp: "Formatted at 2026-09-27 00:50:04 on dist-test-slave-86rs"
server_key: "858c05533e410e04de2055482b692abf"
server_key_iv: "37b66fd31c71c957e6bee4e029eb8982"
server_key_version: "encryptionkey@0"
I20260927 00:50:04.599913 16136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ba6f939d15304d43b4212ea876b03187"
format_stamp: "Formatted at 2026-09-27 00:50:04 on dist-test-slave-86rs"
server_key: "858c05533e410e04de2055482b692abf"
server_key_iv: "37b66fd31c71c957e6bee4e029eb8982"
server_key_version: "encryptionkey@0"
I20260927 00:50:04.604823 16136 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 00:50:04.608364 16151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:50:04.610345 16136 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:50:04.610496 16136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "ba6f939d15304d43b4212ea876b03187"
format_stamp: "Formatted at 2026-09-27 00:50:04 on dist-test-slave-86rs"
server_key: "858c05533e410e04de2055482b692abf"
server_key_iv: "37b66fd31c71c957e6bee4e029eb8982"
server_key_version: "encryptionkey@0"
I20260927 00:50:04.610635 16136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:50:04.630102 16136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:50:04.635586 16136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:50:04.635874 16136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:50:04.637038 16136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:50:04.638808 16136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:50:04.638895 16136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:50:04.638978 16136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:50:04.639014 16136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:50:04.653025 16136 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33475
I20260927 00:50:04.653048 16264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33475 every 8 connection(s)
I20260927 00:50:04.654712 16136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:50:04.656911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 16136
I20260927 00:50:04.657064 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:50:04.657477 392 external_mini_cluster.cc:1428] Setting key afa62f79146b242ef40a7f6201430095
Sep 27 00:50:04 dist-test-slave-86rs krb5kdc[15761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:50:04.677282 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:04.660383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41897 (local address 127.0.98.62:34319)
0927 00:50:04.660557 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 00:50:04.660564 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:50:04.660585 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:50:04.661861 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:50:04.661866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:50:04.661962 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:50:04.662097 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:50:04.663948 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.665031 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.667403 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.667864 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.668040 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:04.672444 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:50:04.672451 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:50:04.672457 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:50:04.672504 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:50:04.674587 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:50:04.675288 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:50:04.675294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:50:04.675296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:50:04.675403 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:50:04.675823 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:50:04.675829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:50:04.675831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:50:04.676080 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:50:04.676207 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 00:50:04.676911 (+ 704us) server_negotiation.cc:299] Negotiation successful
0927 00:50:04.677065 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 00:50:04.678560 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:04.658461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.3:41897)
0927 00:50:04.659231 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:04.659302 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 00:50:04.661450 (+ 2148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:50:04.662424 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:50:04.662435 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:50:04.662992 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:50:04.663716 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:04.663749 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.665242 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.665247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:04.667186 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:04.667205 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.668126 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.668130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:04.668466 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:04.669349 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:50:04.669376 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:50:04.671975 (+ 2599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:50:04.674770 (+ 2795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:50:04.674780 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:50:04.674811 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:50:04.675154 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:50:04.675518 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:50:04.675522 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:50:04.675524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:50:04.675714 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:50:04.676217 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:50:04.676255 (+ 38us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:50:04.676612 (+ 357us) client_negotiation.cc:769] Sending connection context
0927 00:50:04.676869 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 00:50:04.677333 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":210,"threads_started":1}
I20260927 00:50:04.680342 16265 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34319
I20260927 00:50:04.680749 16265 heartbeater.cc:464] Registering TS with master...
I20260927 00:50:04.681478 16265 heartbeater.cc:511] Master 127.0.98.62:34319 requested a full tablet report, sending...
I20260927 00:50:04.683028 15806 ts_manager.cc:194] Registered new tserver with Master: ba6f939d15304d43b4212ea876b03187 (127.0.98.3:33475)
I20260927 00:50:04.683746 15806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41897
I20260927 00:50:04.685587 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:50:04.697670 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:04.689054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43004 (local address 127.0.98.62:34319)
0927 00:50:04.689254 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 00:50:04.689262 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:50:04.689481 (+ 219us) server_negotiation.cc:407] Connection header received
0927 00:50:04.689597 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:50:04.689603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:50:04.689681 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:50:04.689785 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:50:04.690591 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.691242 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.692193 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.692506 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.692699 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:04.694226 (+ 1527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:50:04.694231 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:50:04.694234 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:50:04.694264 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:50:04.695787 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:50:04.696259 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:50:04.696267 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:50:04.696268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:50:04.696388 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:50:04.696681 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:50:04.696684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:50:04.696686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:50:04.696945 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:50:04.697098 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 00:50:04.697346 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 00:50:04.697475 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 00:50:04.701268 15806 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:50:04.705132 15806 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:50:04.725564 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:04.717739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43741 (local address 127.0.0.1:43826)
0927 00:50:04.718415 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:04.718433 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:50:04.718536 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:50:04.719553 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:50:04.719560 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:50:04.719585 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:50:04.719773 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:04.719786 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.721567 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.721571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:04.723794 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:04.723804 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.725130 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.725133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:04.725238 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:04.725261 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:50:04.725330 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 00:50:04.725398 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":72,"thread_start_us":107,"threads_started":1}
I20260927 00:50:04.725746 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:04.718240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43826 (local address 127.0.98.2:43741)
0927 00:50:04.718665 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0927 00:50:04.718683 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:50:04.718713 (+ 30us) server_negotiation.cc:407] Connection header received
0927 00:50:04.718817 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:50:04.718823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:50:04.719107 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:50:04.719318 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:50:04.719922 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.721397 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.723935 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.724817 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.724914 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:04.725073 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 00:50:04.725405 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 00:50:04.725554 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":121,"threads_started":1}
I20260927 00:50:04.726274 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:04.718739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55376 (local address 127.0.98.3:33475)
0927 00:50:04.719652 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0927 00:50:04.719665 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:50:04.719691 (+ 26us) server_negotiation.cc:407] Connection header received
0927 00:50:04.719781 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:50:04.719787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:50:04.719971 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:50:04.720437 (+ 466us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:50:04.720615 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.721748 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.723545 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.724557 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.724966 (+ 409us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:04.725136 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 00:50:04.725986 (+ 850us) server_negotiation.cc:299] Negotiation successful
0927 00:50:04.726105 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":795,"thread_start_us":156,"threads_started":1}
I20260927 00:50:04.725800 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:04.716901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33475 (local address 127.0.0.1:55376)
0927 00:50:04.717338 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:04.717405 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 00:50:04.717567 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:50:04.720132 (+ 2565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:50:04.720135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:50:04.720153 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:50:04.720292 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:04.720299 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.721908 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.721911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:04.723379 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:04.723391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.724720 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.724724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:04.725442 (+ 718us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:04.725467 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:50:04.725544 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 00:50:04.725614 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":567,"thread_start_us":120,"threads_started":1}
I20260927 00:50:04.725888 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:04.717374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40125 (local address 127.0.0.1:36720)
0927 00:50:04.718013 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:04.718031 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:50:04.718137 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:50:04.719806 (+ 1669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:50:04.719812 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:50:04.719836 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:50:04.719957 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:04.719966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.721721 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.721724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:04.723379 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:04.723391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.724720 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.724724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:04.724938 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:04.724989 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 00:50:04.725110 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 00:50:04.725284 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":209,"thread_start_us":138,"threads_started":1}
I20260927 00:50:04.727418 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:04.717883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36720 (local address 127.0.98.1:40125)
0927 00:50:04.719269 (+ 1386us) server_negotiation.cc:206] Beginning negotiation
0927 00:50:04.719276 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:50:04.719305 (+ 29us) server_negotiation.cc:407] Connection header received
0927 00:50:04.719400 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:50:04.719405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:50:04.719629 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:50:04.720129 (+ 500us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:50:04.720336 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.721572 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.723544 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.724557 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.726934 (+ 2377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:04.727099 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 00:50:04.727151 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 00:50:04.727268 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1224,"thread_start_us":164,"threads_started":1}
I20260927 00:50:04.730737 16199 tablet_service.cc:1467] Processing CreateTablet for tablet 952f94c46e2642a59d7890facb96a2de (DEFAULT_TABLE table=test-table [id=5366a0de2c6946a5b96f83c485a1775e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:50:04.730741 15931 tablet_service.cc:1467] Processing CreateTablet for tablet 952f94c46e2642a59d7890facb96a2de (DEFAULT_TABLE table=test-table [id=5366a0de2c6946a5b96f83c485a1775e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:50:04.730741 16065 tablet_service.cc:1467] Processing CreateTablet for tablet 952f94c46e2642a59d7890facb96a2de (DEFAULT_TABLE table=test-table [id=5366a0de2c6946a5b96f83c485a1775e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:50:04.732134 16199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 952f94c46e2642a59d7890facb96a2de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:50:04.732132 16065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 952f94c46e2642a59d7890facb96a2de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:50:04.732193 15931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 952f94c46e2642a59d7890facb96a2de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:50:04.741366 16282 tablet_bootstrap.cc:492] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a: Bootstrap starting.
I20260927 00:50:04.742662 16284 tablet_bootstrap.cc:492] T 952f94c46e2642a59d7890facb96a2de P 5dd062a9695e41c59a75ef425fb7deed: Bootstrap starting.
I20260927 00:50:04.743116 16283 tablet_bootstrap.cc:492] T 952f94c46e2642a59d7890facb96a2de P ba6f939d15304d43b4212ea876b03187: Bootstrap starting.
I20260927 00:50:04.745265 16283 tablet_bootstrap.cc:654] T 952f94c46e2642a59d7890facb96a2de P ba6f939d15304d43b4212ea876b03187: Neither blocks nor log segments found. Creating new log.
I20260927 00:50:04.745294 16282 tablet_bootstrap.cc:654] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a: Neither blocks nor log segments found. Creating new log.
I20260927 00:50:04.746357 16283 log.cc:824] T 952f94c46e2642a59d7890facb96a2de P ba6f939d15304d43b4212ea876b03187: Log is configured to *not* fsync() on all Append() calls
I20260927 00:50:04.746363 16282 log.cc:824] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a: Log is configured to *not* fsync() on all Append() calls
I20260927 00:50:04.747102 16284 tablet_bootstrap.cc:654] T 952f94c46e2642a59d7890facb96a2de P 5dd062a9695e41c59a75ef425fb7deed: Neither blocks nor log segments found. Creating new log.
I20260927 00:50:04.747865 16284 log.cc:824] T 952f94c46e2642a59d7890facb96a2de P 5dd062a9695e41c59a75ef425fb7deed: Log is configured to *not* fsync() on all Append() calls
I20260927 00:50:04.748786 16283 tablet_bootstrap.cc:492] T 952f94c46e2642a59d7890facb96a2de P ba6f939d15304d43b4212ea876b03187: No bootstrap required, opened a new log
I20260927 00:50:04.749038 16283 ts_tablet_manager.cc:1397] T 952f94c46e2642a59d7890facb96a2de P ba6f939d15304d43b4212ea876b03187: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:50:04.749557 16282 tablet_bootstrap.cc:492] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a: No bootstrap required, opened a new log
I20260927 00:50:04.749770 16282 ts_tablet_manager.cc:1397] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 00:50:04.750360 16284 tablet_bootstrap.cc:492] T 952f94c46e2642a59d7890facb96a2de P 5dd062a9695e41c59a75ef425fb7deed: No bootstrap required, opened a new log
I20260927 00:50:04.750582 16284 ts_tablet_manager.cc:1397] T 952f94c46e2642a59d7890facb96a2de P 5dd062a9695e41c59a75ef425fb7deed: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 00:50:04.752921 16283 raft_consensus.cc:348] T 952f94c46e2642a59d7890facb96a2de P ba6f939d15304d43b4212ea876b03187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 } } peers { permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 } } peers { permanent_uuid: "a52f4e455782487687acb63ecff91e1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260927 00:50:04.753227 16283 raft_consensus.cc:374] T 952f94c46e2642a59d7890facb96a2de P ba6f939d15304d43b4212ea876b03187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:50:04.753304 16283 raft_consensus.cc:729] T 952f94c46e2642a59d7890facb96a2de P ba6f939d15304d43b4212ea876b03187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba6f939d15304d43b4212ea876b03187, State: Initialized, Role: FOLLOWER
I20260927 00:50:04.753839 16283 consensus_queue.cc:260] T 952f94c46e2642a59d7890facb96a2de P ba6f939d15304d43b4212ea876b03187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 } } peers { permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 } } peers { permanent_uuid: "a52f4e455782487687acb63ecff91e1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260927 00:50:04.754594 16265 heartbeater.cc:503] Master 127.0.98.62:34319 was elected leader, sending a full tablet report...
I20260927 00:50:04.755040 16283 ts_tablet_manager.cc:1428] T 952f94c46e2642a59d7890facb96a2de P ba6f939d15304d43b4212ea876b03187: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260927 00:50:04.754886 16282 raft_consensus.cc:348] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 } } peers { permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 } } peers { permanent_uuid: "a52f4e455782487687acb63ecff91e1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260927 00:50:04.754959 16284 raft_consensus.cc:348] T 952f94c46e2642a59d7890facb96a2de P 5dd062a9695e41c59a75ef425fb7deed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 } } peers { permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 } } peers { permanent_uuid: "a52f4e455782487687acb63ecff91e1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260927 00:50:04.755177 16282 raft_consensus.cc:374] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:50:04.755239 16284 raft_consensus.cc:374] T 952f94c46e2642a59d7890facb96a2de P 5dd062a9695e41c59a75ef425fb7deed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:50:04.755327 16282 raft_consensus.cc:729] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a52f4e455782487687acb63ecff91e1a, State: Initialized, Role: FOLLOWER
I20260927 00:50:04.755386 16284 raft_consensus.cc:729] T 952f94c46e2642a59d7890facb96a2de P 5dd062a9695e41c59a75ef425fb7deed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd062a9695e41c59a75ef425fb7deed, State: Initialized, Role: FOLLOWER
I20260927 00:50:04.755896 16284 consensus_queue.cc:260] T 952f94c46e2642a59d7890facb96a2de P 5dd062a9695e41c59a75ef425fb7deed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 } } peers { permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 } } peers { permanent_uuid: "a52f4e455782487687acb63ecff91e1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260927 00:50:04.756099 16282 consensus_queue.cc:260] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 } } peers { permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 } } peers { permanent_uuid: "a52f4e455782487687acb63ecff91e1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260927 00:50:04.756804 16131 heartbeater.cc:503] Master 127.0.98.62:34319 was elected leader, sending a full tablet report...
I20260927 00:50:04.757186 16284 ts_tablet_manager.cc:1428] T 952f94c46e2642a59d7890facb96a2de P 5dd062a9695e41c59a75ef425fb7deed: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 00:50:04.758440 16282 ts_tablet_manager.cc:1428] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a: Time spent starting tablet: real 0.009s user 0.003s sys 0.003s
I20260927 00:50:04.758842 15997 heartbeater.cc:503] Master 127.0.98.62:34319 was elected leader, sending a full tablet report...
W20260927 00:50:04.838244 15998 tablet.cc:2369] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:50:04.850476 16290 raft_consensus.cc:482] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:50:04.850706 16290 raft_consensus.cc:504] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 } } peers { permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 } } peers { permanent_uuid: "a52f4e455782487687acb63ecff91e1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260927 00:50:04.851963 16290 leader_election.cc:288] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba6f939d15304d43b4212ea876b03187 (127.0.98.3:33475), 5dd062a9695e41c59a75ef425fb7deed (127.0.98.2:43741)
I20260927 00:50:04.856911 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:04.853101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46199 (local address 127.0.98.2:43741)
0927 00:50:04.853263 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 00:50:04.853267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:50:04.853416 (+ 149us) server_negotiation.cc:407] Connection header received
0927 00:50:04.853522 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:50:04.853526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:50:04.853577 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:50:04.853671 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:50:04.854012 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.854600 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.855822 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.856285 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.856415 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:04.856513 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:50:04.856680 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 00:50:04.856761 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 00:50:04.856922 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:04.852426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33475 (local address 127.0.98.1:42407)
0927 00:50:04.852830 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:04.852879 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 00:50:04.853056 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:50:04.853450 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:50:04.853454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:50:04.853478 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:50:04.853657 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:04.853667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.854556 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.854560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:04.855627 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:04.855637 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.856483 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.856487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:04.856602 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:04.856624 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:50:04.856706 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 00:50:04.856774 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":178,"threads_started":1}
I20260927 00:50:04.856942 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:04.852461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42407 (local address 127.0.98.3:33475)
0927 00:50:04.852653 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 00:50:04.852659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:50:04.853184 (+ 525us) server_negotiation.cc:407] Connection header received
0927 00:50:04.853255 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:50:04.853258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:50:04.853324 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:50:04.853409 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:50:04.853792 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.854429 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.855754 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.856326 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.856479 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:04.856548 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 00:50:04.856746 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 00:50:04.856821 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 00:50:04.856928 16292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:04.852891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43741 (local address 127.0.98.1:46199)
0927 00:50:04.853294 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:04.853307 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:50:04.853421 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:50:04.853703 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:50:04.853706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:50:04.853754 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:50:04.853871 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:04.853877 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.854730 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.854733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:04.855662 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:04.855671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.856422 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.856428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:04.856532 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:04.856560 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 00:50:04.856641 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 00:50:04.856739 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":102,"threads_started":1}
I20260927 00:50:04.857673 16219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "952f94c46e2642a59d7890facb96a2de" candidate_uuid: "a52f4e455782487687acb63ecff91e1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba6f939d15304d43b4212ea876b03187" is_pre_election: true
I20260927 00:50:04.857709 16085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "952f94c46e2642a59d7890facb96a2de" candidate_uuid: "a52f4e455782487687acb63ecff91e1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd062a9695e41c59a75ef425fb7deed" is_pre_election: true
I20260927 00:50:04.858013 16219 raft_consensus.cc:2454] T 952f94c46e2642a59d7890facb96a2de P ba6f939d15304d43b4212ea876b03187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a52f4e455782487687acb63ecff91e1a in term 0.
I20260927 00:50:04.858016 16085 raft_consensus.cc:2454] T 952f94c46e2642a59d7890facb96a2de P 5dd062a9695e41c59a75ef425fb7deed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a52f4e455782487687acb63ecff91e1a in term 0.
I20260927 00:50:04.858531 15885 leader_election.cc:302] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd062a9695e41c59a75ef425fb7deed, a52f4e455782487687acb63ecff91e1a; no voters:
I20260927 00:50:04.858850 16290 raft_consensus.cc:2781] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:50:04.858938 16290 raft_consensus.cc:482] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:50:04.858969 16290 raft_consensus.cc:3037] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:50:04.860100 16290 raft_consensus.cc:504] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 } } peers { permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 } } peers { permanent_uuid: "a52f4e455782487687acb63ecff91e1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260927 00:50:04.860498 16290 leader_election.cc:288] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a [CANDIDATE]: Term 1 election: Requested vote from peers ba6f939d15304d43b4212ea876b03187 (127.0.98.3:33475), 5dd062a9695e41c59a75ef425fb7deed (127.0.98.2:43741)
I20260927 00:50:04.860899 16219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "952f94c46e2642a59d7890facb96a2de" candidate_uuid: "a52f4e455782487687acb63ecff91e1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba6f939d15304d43b4212ea876b03187"
I20260927 00:50:04.861044 16219 raft_consensus.cc:3037] T 952f94c46e2642a59d7890facb96a2de P ba6f939d15304d43b4212ea876b03187 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:50:04.861199 16085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "952f94c46e2642a59d7890facb96a2de" candidate_uuid: "a52f4e455782487687acb63ecff91e1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd062a9695e41c59a75ef425fb7deed"
I20260927 00:50:04.861337 16085 raft_consensus.cc:3037] T 952f94c46e2642a59d7890facb96a2de P 5dd062a9695e41c59a75ef425fb7deed [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:50:04.862550 16219 raft_consensus.cc:2454] T 952f94c46e2642a59d7890facb96a2de P ba6f939d15304d43b4212ea876b03187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a52f4e455782487687acb63ecff91e1a in term 1.
I20260927 00:50:04.862658 16085 raft_consensus.cc:2454] T 952f94c46e2642a59d7890facb96a2de P 5dd062a9695e41c59a75ef425fb7deed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a52f4e455782487687acb63ecff91e1a in term 1.
I20260927 00:50:04.862965 15885 leader_election.cc:302] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a52f4e455782487687acb63ecff91e1a, ba6f939d15304d43b4212ea876b03187; no voters:
I20260927 00:50:04.863193 16290 raft_consensus.cc:2781] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:50:04.863457 16290 raft_consensus.cc:686] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a [term 1 LEADER]: Becoming Leader. State: Replica: a52f4e455782487687acb63ecff91e1a, State: Running, Role: LEADER
I20260927 00:50:04.863827 16290 consensus_queue.cc:237] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 } } peers { permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 } } peers { permanent_uuid: "a52f4e455782487687acb63ecff91e1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260927 00:50:04.867470 15804 catalog_manager.cc:5495] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a reported cstate change: term changed from 0 to 1, leader changed from <none> to a52f4e455782487687acb63ecff91e1a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a52f4e455782487687acb63ecff91e1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a52f4e455782487687acb63ecff91e1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } health_report { overall_health: HEALTHY } } }
I20260927 00:50:04.882184 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:04.878383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36722 (local address 127.0.98.1:40125)
0927 00:50:04.878502 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 00:50:04.878505 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:50:04.878519 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:50:04.878632 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:50:04.878634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:50:04.878693 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:50:04.878788 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:50:04.879077 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.879685 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.880629 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:04.880864 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:04.880966 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:04.881252 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:50:04.881762 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:50:04.881862 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 00:50:04.881923 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 00:50:04.881979 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 00:50:04.889392 16085 raft_consensus.cc:1261] T 952f94c46e2642a59d7890facb96a2de P 5dd062a9695e41c59a75ef425fb7deed [term 1 FOLLOWER]: Refusing update from remote peer a52f4e455782487687acb63ecff91e1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:50:04.889400 16219 raft_consensus.cc:1261] T 952f94c46e2642a59d7890facb96a2de P ba6f939d15304d43b4212ea876b03187 [term 1 FOLLOWER]: Refusing update from remote peer a52f4e455782487687acb63ecff91e1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 00:50:04.889659 16132 tablet.cc:2369] T 952f94c46e2642a59d7890facb96a2de P 5dd062a9695e41c59a75ef425fb7deed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:50:04.890252 16290 consensus_queue.cc:1035] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:50:04.890462 16293 consensus_queue.cc:1035] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:50:04.898797 16296 mvcc.cc:205] Tried to move back new op lower bound from 7333765959217532928 to 7333765959125270528. Current Snapshot: MvccSnapshot[applied={T|T < 7333765959217532928}]
I20260927 00:50:04.900453 16297 mvcc.cc:205] Tried to move back new op lower bound from 7333765959217532928 to 7333765959125270528. Current Snapshot: MvccSnapshot[applied={T|T < 7333765959217532928}]
I20260927 00:50:04.909065 15804 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:50:04.909308 15804 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:50:04.912048 15804 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1: Sending DeleteTablet for 3 replicas of tablet 952f94c46e2642a59d7890facb96a2de
I20260927 00:50:04.913206 15931 tablet_service.cc:1514] Processing DeleteTablet for tablet 952f94c46e2642a59d7890facb96a2de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:50:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36720
I20260927 00:50:04.913208 16199 tablet_service.cc:1514] Processing DeleteTablet for tablet 952f94c46e2642a59d7890facb96a2de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:50:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55376
I20260927 00:50:04.913337 16065 tablet_service.cc:1514] Processing DeleteTablet for tablet 952f94c46e2642a59d7890facb96a2de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:50:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43826
I20260927 00:50:04.913874 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 15868
I20260927 00:50:04.915026 16308 tablet_replica.cc:330] T 952f94c46e2642a59d7890facb96a2de P 5dd062a9695e41c59a75ef425fb7deed: stopping tablet replica
I20260927 00:50:04.915438 16308 raft_consensus.cc:2229] T 952f94c46e2642a59d7890facb96a2de P 5dd062a9695e41c59a75ef425fb7deed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:50:04.915752 16308 raft_consensus.cc:2258] T 952f94c46e2642a59d7890facb96a2de P 5dd062a9695e41c59a75ef425fb7deed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:50:04.917308 16308 ts_tablet_manager.cc:1905] T 952f94c46e2642a59d7890facb96a2de P 5dd062a9695e41c59a75ef425fb7deed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:50:04.919692 16308 ts_tablet_manager.cc:1918] T 952f94c46e2642a59d7890facb96a2de P 5dd062a9695e41c59a75ef425fb7deed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:50:04.919782 16308 log.cc:1196] T 952f94c46e2642a59d7890facb96a2de P 5dd062a9695e41c59a75ef425fb7deed: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/wal/wals/952f94c46e2642a59d7890facb96a2de
I20260927 00:50:04.920045 16307 tablet_replica.cc:330] T 952f94c46e2642a59d7890facb96a2de P ba6f939d15304d43b4212ea876b03187: stopping tablet replica
I20260927 00:50:04.920104 16308 ts_tablet_manager.cc:1939] T 952f94c46e2642a59d7890facb96a2de P 5dd062a9695e41c59a75ef425fb7deed: Deleting consensus metadata
I20260927 00:50:04.920452 16307 raft_consensus.cc:2229] T 952f94c46e2642a59d7890facb96a2de P ba6f939d15304d43b4212ea876b03187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:50:04.920809 16307 raft_consensus.cc:2258] T 952f94c46e2642a59d7890facb96a2de P ba6f939d15304d43b4212ea876b03187 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 00:50:04.921572 15790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40125 (error 108)
I20260927 00:50:04.921638 15791 catalog_manager.cc:4841] TS 5dd062a9695e41c59a75ef425fb7deed (127.0.98.2:43741): tablet 952f94c46e2642a59d7890facb96a2de (table test-table [id=5366a0de2c6946a5b96f83c485a1775e]) successfully deleted
I20260927 00:50:04.922518 16307 ts_tablet_manager.cc:1905] T 952f94c46e2642a59d7890facb96a2de P ba6f939d15304d43b4212ea876b03187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:50:04.922574 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:04.922288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40125 (local address 127.0.0.1:36734)
0927 00:50:04.922422 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:04.922492 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 00:50:04.922739 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 16002
W20260927 00:50:04.922777 15790 catalog_manager.cc:4584] TS a52f4e455782487687acb63ecff91e1a (127.0.98.1:40125): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 952f94c46e2642a59d7890facb96a2de: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40125: connect: Connection refused (error 111)
I20260927 00:50:04.925714 16307 ts_tablet_manager.cc:1918] T 952f94c46e2642a59d7890facb96a2de P ba6f939d15304d43b4212ea876b03187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:50:04.925799 16307 log.cc:1196] T 952f94c46e2642a59d7890facb96a2de P ba6f939d15304d43b4212ea876b03187: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/wal/wals/952f94c46e2642a59d7890facb96a2de
I20260927 00:50:04.926148 16307 ts_tablet_manager.cc:1939] T 952f94c46e2642a59d7890facb96a2de P ba6f939d15304d43b4212ea876b03187: Deleting consensus metadata
I20260927 00:50:04.927417 15791 catalog_manager.cc:4841] TS ba6f939d15304d43b4212ea876b03187 (127.0.98.3:33475): tablet 952f94c46e2642a59d7890facb96a2de (table test-table [id=5366a0de2c6946a5b96f83c485a1775e]) successfully deleted
I20260927 00:50:04.929996 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 16136
I20260927 00:50:04.936869 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 15774
I20260927 00:50:24.944031 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34319
--webserver_interface=127.0.98.62
--webserver_port=41639
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:50:25.071318 16310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:50:25.071780 16310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:50:25.071863 16310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:50:25.076756 16310 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 00:50:25.076831 16310 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 00:50:25.076848 16310 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 00:50:25.076864 16310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:50:25.076884 16310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:50:25.076902 16310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:50:25.076964 16310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:50:25.076992 16310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:50:25.082974 16310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:50:25.084364 16310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:50:25.085556 16310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:50:25.100666 16310 server_base.cc:1034] running on GCE node
W20260927 00:50:25.100596 16317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:50:25.100589 16316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:50:25.100611 16319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:50:25.101706 16310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:50:25.103071 16310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:50:25.104298 16310 hybrid_clock.cc:648] HybridClock initialized: now 1790470225104279 us; error 28 us; skew 500 ppm
Sep 27 00:50:25 dist-test-slave-86rs krb5kdc[15761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:50:25.113517 16310 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:50:25.114668 16310 webserver.cc:460] Webserver started at http://127.0.98.62:41639/ using document root <none> and password file <none>
I20260927 00:50:25.115360 16310 fs_manager.cc:359] Metadata directory not provided
I20260927 00:50:25.115473 16310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:50:25.119993 16310 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260927 00:50:25.122527 16326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:50:25.124019 16310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:50:25.124193 16310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "f491678175874e80b10a2e572a51ccd1"
format_stamp: "Formatted at 2026-09-27 00:50:03 on dist-test-slave-86rs"
server_key: "76045a0732c9276322172d0aa3f8cc2e"
server_key_iv: "b8cc60c006346ca8afe1a7ff8ac77b7b"
server_key_version: "encryptionkey@0"
I20260927 00:50:25.124984 16310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:50:25.137971 16310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:50:25.141575 16310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:50:25.141809 16310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:50:25.150069 16310 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34319
I20260927 00:50:25.150374 16378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34319 every 8 connection(s)
I20260927 00:50:25.151419 16310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:50:25.156523 16379 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:50:25.158525 16379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1: Bootstrap starting.
I20260927 00:50:25.161751 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 16310
I20260927 00:50:25.162307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40125
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34319
--webserver_port=45759
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:50:25.177786 16379 log.cc:824] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1: Log is configured to *not* fsync() on all Append() calls
I20260927 00:50:25.187453 16379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:50:25.187980 16379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1: Bootstrap complete.
I20260927 00:50:25.192448 16379 raft_consensus.cc:348] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f491678175874e80b10a2e572a51ccd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } }
I20260927 00:50:25.192926 16379 raft_consensus.cc:729] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f491678175874e80b10a2e572a51ccd1, State: Initialized, Role: FOLLOWER
I20260927 00:50:25.193539 16379 consensus_queue.cc:260] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f491678175874e80b10a2e572a51ccd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } }
I20260927 00:50:25.193665 16379 raft_consensus.cc:388] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:50:25.193758 16379 raft_consensus.cc:482] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:50:25.193862 16379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:50:25.196098 16379 raft_consensus.cc:504] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f491678175874e80b10a2e572a51ccd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } }
I20260927 00:50:25.196524 16379 leader_election.cc:302] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f491678175874e80b10a2e572a51ccd1; no voters:
I20260927 00:50:25.196909 16379 leader_election.cc:288] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 00:50:25.197000 16383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 00:50:25.197208 16383 raft_consensus.cc:686] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [term 2 LEADER]: Becoming Leader. State: Replica: f491678175874e80b10a2e572a51ccd1, State: Running, Role: LEADER
I20260927 00:50:25.197593 16383 consensus_queue.cc:237] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f491678175874e80b10a2e572a51ccd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } }
I20260927 00:50:25.198238 16379 sys_catalog.cc:564] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:50:25.198658 16383 sys_catalog.cc:455] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f491678175874e80b10a2e572a51ccd1. Latest consensus state: current_term: 2 leader_uuid: "f491678175874e80b10a2e572a51ccd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f491678175874e80b10a2e572a51ccd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } } }
I20260927 00:50:25.198781 16383 sys_catalog.cc:458] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [sys.catalog]: This master's current role is: LEADER
I20260927 00:50:25.199224 16384 sys_catalog.cc:455] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f491678175874e80b10a2e572a51ccd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f491678175874e80b10a2e572a51ccd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } } }
I20260927 00:50:25.199393 16384 sys_catalog.cc:458] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [sys.catalog]: This master's current role is: LEADER
I20260927 00:50:25.200654 16387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:50:25.205884 16387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:50:25.206601 16387 catalog_manager.cc:1259] Loaded cluster ID: f18e887dadde4191a8a38af6fa42730e
I20260927 00:50:25.206671 16387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:50:25.209317 16387 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:50:25.210076 16387 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1: Loaded TSK: 0
I20260927 00:50:25.225955 16387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1: Generated new TSK 1
I20260927 00:50:25.226282 16387 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:50:25.285394 16381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:50:25.285775 16381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:50:25.285888 16381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:50:25.290488 16381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:50:25.290607 16381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:50:25.290714 16381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:50:25.296953 16381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40125
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45759
--tserver_master_addrs=127.0.98.62:34319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:50:25.298106 16381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:50:25.299292 16381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:50:25.310891 16406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:50:25.310896 16409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:50:25.310885 16407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:50:25.312032 16381 server_base.cc:1034] running on GCE node
I20260927 00:50:25.312585 16381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:50:25.313361 16381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:50:25.314585 16381 hybrid_clock.cc:648] HybridClock initialized: now 1790470225314559 us; error 41 us; skew 500 ppm
Sep 27 00:50:25 dist-test-slave-86rs krb5kdc[15761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:50:25.317754 16381 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:50:25.318835 16381 webserver.cc:460] Webserver started at http://127.0.98.1:45759/ using document root <none> and password file <none>
I20260927 00:50:25.319509 16381 fs_manager.cc:359] Metadata directory not provided
I20260927 00:50:25.319628 16381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:50:25.323727 16381 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:50:25.326205 16416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:50:25.327646 16381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 00:50:25.327756 16381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "a52f4e455782487687acb63ecff91e1a"
format_stamp: "Formatted at 2026-09-27 00:50:04 on dist-test-slave-86rs"
server_key: "a6c3acbb676d08ca37fbc3b5e6ae7f57"
server_key_iv: "4303f9cd97d5b3901cac3b9b76c016cd"
server_key_version: "encryptionkey@0"
I20260927 00:50:25.328480 16381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:50:25.349615 16381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:50:25.353397 16381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:50:25.353641 16381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:50:25.354377 16381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:50:25.355683 16423 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 00:50:25.360059 16381 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 00:50:25.360160 16381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 00:50:25.360291 16381 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 00:50:25.362118 16381 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 00:50:25.362195 16381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 00:50:25.362509 16423 tablet_bootstrap.cc:492] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a: Bootstrap starting.
I20260927 00:50:25.373109 16381 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40125
I20260927 00:50:25.373180 16530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40125 every 8 connection(s)
I20260927 00:50:25.374176 16381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:50:25.380095 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 16381
I20260927 00:50:25.380685 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43741
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34319
--webserver_port=34355
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:50:25.382280 16423 log.cc:824] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:50:25 dist-test-slave-86rs krb5kdc[15761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:50:25.392531 16423 tablet_bootstrap.cc:492] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:50:25.392611 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:25.376459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.1:38361)
0927 00:50:25.376985 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:25.377040 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 00:50:25.377907 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:50:25.379378 (+ 1471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:50:25.379385 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:50:25.379772 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:50:25.380428 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:25.380448 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.381997 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.382003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:25.383531 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:25.383550 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.384429 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.384433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:25.384620 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:25.385301 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:50:25.385319 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:50:25.386900 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:50:25.389246 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:50:25.389252 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:50:25.389266 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:50:25.389591 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:50:25.389974 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:50:25.389977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:50:25.389979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:50:25.390126 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:50:25.390724 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:50:25.390730 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:50:25.391000 (+ 270us) client_negotiation.cc:769] Sending connection context
0927 00:50:25.391193 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 00:50:25.391576 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":145,"threads_started":1}
I20260927 00:50:25.393021 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:25.376495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38361 (local address 127.0.98.62:34319)
0927 00:50:25.377117 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0927 00:50:25.377135 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:50:25.377186 (+ 51us) server_negotiation.cc:407] Connection header received
0927 00:50:25.378433 (+ 1247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:50:25.378449 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:50:25.378897 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:50:25.379326 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:50:25.380648 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.381827 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.383684 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.384177 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.384481 (+ 304us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:25.387121 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:50:25.387127 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:50:25.387140 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:50:25.387169 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:50:25.389093 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:50:25.389722 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:50:25.389736 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:50:25.389741 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:50:25.389872 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:50:25.390224 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:50:25.390229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:50:25.390231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:50:25.390606 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:50:25.390754 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 00:50:25.391311 (+ 557us) server_negotiation.cc:299] Negotiation successful
0927 00:50:25.391587 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":170,"threads_started":1}
I20260927 00:50:25.393079 16423 tablet_bootstrap.cc:492] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a: Bootstrap complete.
I20260927 00:50:25.394675 16423 ts_tablet_manager.cc:1397] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a: Time spent bootstrapping tablet: real 0.032s user 0.025s sys 0.004s
I20260927 00:50:25.395653 16531 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34319
I20260927 00:50:25.395931 16531 heartbeater.cc:464] Registering TS with master...
I20260927 00:50:25.396564 16531 heartbeater.cc:511] Master 127.0.98.62:34319 requested a full tablet report, sending...
I20260927 00:50:25.397913 16423 raft_consensus.cc:348] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 } } peers { permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 } } peers { permanent_uuid: "a52f4e455782487687acb63ecff91e1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260927 00:50:25.398481 16423 raft_consensus.cc:729] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a52f4e455782487687acb63ecff91e1a, State: Initialized, Role: FOLLOWER
I20260927 00:50:25.399132 16423 consensus_queue.cc:260] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 } } peers { permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 } } peers { permanent_uuid: "a52f4e455782487687acb63ecff91e1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260927 00:50:25.399294 16343 ts_manager.cc:194] Registered new tserver with Master: a52f4e455782487687acb63ecff91e1a (127.0.98.1:40125)
I20260927 00:50:25.400352 16423 ts_tablet_manager.cc:1428] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260927 00:50:25.403079 16343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38361
I20260927 00:50:25.405984 16531 heartbeater.cc:503] Master 127.0.98.62:34319 was elected leader, sending a full tablet report...
Sep 27 00:50:25 dist-test-slave-86rs krb5kdc[15761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:50:25.414763 16538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:25.402576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40125 (local address 127.0.0.1:43816)
0927 00:50:25.403462 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:25.403511 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 00:50:25.403649 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:50:25.404502 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:50:25.404510 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:50:25.404990 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:50:25.405132 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:25.405141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.406761 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.406764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:25.407713 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:25.407726 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.408484 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.408509 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:25.408608 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:25.409232 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:50:25.409244 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:50:25.410662 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:50:25.412899 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:50:25.412909 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:50:25.412914 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:50:25.413162 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:50:25.413497 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:50:25.413500 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:50:25.413501 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:50:25.413585 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:50:25.414160 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:50:25.414166 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:50:25.414287 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 00:50:25.414431 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 00:50:25.414618 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":122,"threads_started":1}
I20260927 00:50:25.414904 16539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:25.402678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43816 (local address 127.0.98.1:40125)
0927 00:50:25.403955 (+ 1277us) server_negotiation.cc:206] Beginning negotiation
0927 00:50:25.403970 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:50:25.403996 (+ 26us) server_negotiation.cc:407] Connection header received
0927 00:50:25.404075 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:50:25.404081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:50:25.404325 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:50:25.404468 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:50:25.405274 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.406455 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.407837 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.408374 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.408483 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:25.410810 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:50:25.410822 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:50:25.410836 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:50:25.410872 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:50:25.412759 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:50:25.413282 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:50:25.413288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:50:25.413290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:50:25.413392 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:50:25.413668 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:50:25.413671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:50:25.413673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:50:25.414005 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:50:25.414136 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 00:50:25.414539 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 00:50:25.414724 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1176,"thread_start_us":111,"threads_started":1}
I20260927 00:50:25.415954 16465 tablet_service.cc:1514] Processing DeleteTablet for tablet 952f94c46e2642a59d7890facb96a2de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:50:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43816
I20260927 00:50:25.416985 16540 tablet_replica.cc:330] stopping tablet replica
I20260927 00:50:25.417302 16540 raft_consensus.cc:2229] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:50:25.417578 16540 raft_consensus.cc:2258] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:50:25.418614 16540 ts_tablet_manager.cc:1905] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:50:25.422485 16540 ts_tablet_manager.cc:1918] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:50:25.422588 16540 log.cc:1196] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/wal/wals/952f94c46e2642a59d7890facb96a2de
I20260927 00:50:25.422874 16540 ts_tablet_manager.cc:1939] T 952f94c46e2642a59d7890facb96a2de P a52f4e455782487687acb63ecff91e1a: Deleting consensus metadata
I20260927 00:50:25.424865 16327 catalog_manager.cc:4841] TS a52f4e455782487687acb63ecff91e1a (127.0.98.1:40125): tablet 952f94c46e2642a59d7890facb96a2de (table test-table [id=5366a0de2c6946a5b96f83c485a1775e]) successfully deleted
W20260927 00:50:25.507465 16535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:50:25.507794 16535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:50:25.507867 16535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:50:25.512187 16535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:50:25.512429 16535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:50:25.512598 16535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:50:25.518914 16535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34355
--tserver_master_addrs=127.0.98.62:34319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:50:25.520143 16535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:50:25.521250 16535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:50:25.531500 16545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:50:25.531558 16548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:50:25.531517 16546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:50:25.531862 16535 server_base.cc:1034] running on GCE node
I20260927 00:50:25.532471 16535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:50:25.533244 16535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:50:25.534492 16535 hybrid_clock.cc:648] HybridClock initialized: now 1790470225534472 us; error 31 us; skew 500 ppm
Sep 27 00:50:25 dist-test-slave-86rs krb5kdc[15761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:50:25.537664 16535 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:50:25.538815 16535 webserver.cc:460] Webserver started at http://127.0.98.2:34355/ using document root <none> and password file <none>
I20260927 00:50:25.539435 16535 fs_manager.cc:359] Metadata directory not provided
I20260927 00:50:25.539513 16535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:50:25.543742 16535 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:50:25.546072 16555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:50:25.547472 16535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:50:25.547617 16535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "5dd062a9695e41c59a75ef425fb7deed"
format_stamp: "Formatted at 2026-09-27 00:50:04 on dist-test-slave-86rs"
server_key: "e54e64010f47e010a8da134e7fe16a4c"
server_key_iv: "67e542ec782de4bf0e996c80619b0aa5"
server_key_version: "encryptionkey@0"
I20260927 00:50:25.548188 16535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:50:25.564950 16535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:50:25.568449 16535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:50:25.568672 16535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:50:25.569377 16535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:50:25.570257 16535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:50:25.570307 16535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:50:25.570379 16535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:50:25.570412 16535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:50:25.580286 16535 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43741
I20260927 00:50:25.580312 16668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43741 every 8 connection(s)
I20260927 00:50:25.581418 16535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:50:25.589578 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 16535
I20260927 00:50:25.590070 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33475
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34319
--webserver_port=38461
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:50:25 dist-test-slave-86rs krb5kdc[15761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:50:25.597512 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:25.583374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59815 (local address 127.0.98.62:34319)
0927 00:50:25.583534 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 00:50:25.583538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:50:25.584390 (+ 852us) server_negotiation.cc:407] Connection header received
0927 00:50:25.585769 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:50:25.585773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:50:25.585847 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:50:25.585926 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:50:25.587270 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.587837 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.589100 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.589472 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.589598 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:25.592561 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:50:25.592565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:50:25.592567 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:50:25.592609 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:50:25.594556 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:50:25.595162 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:50:25.595168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:50:25.595173 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:50:25.595283 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:50:25.595663 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:50:25.595668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:50:25.595671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:50:25.595974 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:50:25.596093 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 00:50:25.597058 (+ 965us) server_negotiation.cc:299] Negotiation successful
0927 00:50:25.597246 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:50:25.598310 16671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:25.583692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.2:59815)
0927 00:50:25.584150 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:25.584195 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 00:50:25.585526 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:50:25.586111 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:50:25.586118 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:50:25.586519 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:50:25.587098 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:25.587116 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.587990 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.587996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:25.588952 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:25.588963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.589710 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.589716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:25.589915 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:25.590687 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:50:25.590704 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:50:25.592373 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:50:25.594704 (+ 2331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:50:25.594709 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:50:25.594730 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:50:25.595027 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:50:25.595394 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:50:25.595397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:50:25.595398 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:50:25.595548 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:50:25.596189 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:50:25.596195 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:50:25.596768 (+ 573us) client_negotiation.cc:769] Sending connection context
0927 00:50:25.596945 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 00:50:25.597243 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":116,"threads_started":1}
I20260927 00:50:25.599686 16669 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34319
I20260927 00:50:25.599972 16669 heartbeater.cc:464] Registering TS with master...
I20260927 00:50:25.600600 16669 heartbeater.cc:511] Master 127.0.98.62:34319 requested a full tablet report, sending...
I20260927 00:50:25.601729 16343 ts_manager.cc:194] Registered new tserver with Master: 5dd062a9695e41c59a75ef425fb7deed (127.0.98.2:43741)
I20260927 00:50:25.602330 16343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59815
W20260927 00:50:25.708861 16672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:50:25.709156 16672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:50:25.709201 16672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:50:25.713451 16672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:50:25.713523 16672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:50:25.713621 16672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:50:25.719365 16672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33475
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38461
--tserver_master_addrs=127.0.98.62:34319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:50:25.720587 16672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:50:25.721637 16672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:50:25.732424 16677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:50:25.732403 16678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:50:25.732564 16672 server_base.cc:1034] running on GCE node
W20260927 00:50:25.732405 16680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:50:25.733362 16672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:50:25.734148 16672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:50:25.735397 16672 hybrid_clock.cc:648] HybridClock initialized: now 1790470225735378 us; error 34 us; skew 500 ppm
Sep 27 00:50:25 dist-test-slave-86rs krb5kdc[15761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:50:25.738473 16672 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:50:25.739392 16672 webserver.cc:460] Webserver started at http://127.0.98.3:38461/ using document root <none> and password file <none>
I20260927 00:50:25.739934 16672 fs_manager.cc:359] Metadata directory not provided
I20260927 00:50:25.740005 16672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:50:25.744067 16672 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:50:25.746351 16687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:50:25.747840 16672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:50:25.747964 16672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "ba6f939d15304d43b4212ea876b03187"
format_stamp: "Formatted at 2026-09-27 00:50:04 on dist-test-slave-86rs"
server_key: "858c05533e410e04de2055482b692abf"
server_key_iv: "37b66fd31c71c957e6bee4e029eb8982"
server_key_version: "encryptionkey@0"
I20260927 00:50:25.748673 16672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:50:25.761520 16672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:50:25.765007 16672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:50:25.765189 16672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:50:25.765777 16672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:50:25.766624 16672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:50:25.766678 16672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:50:25.766721 16672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:50:25.766737 16672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:50:25.775856 16672 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33475
I20260927 00:50:25.775871 16800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33475 every 8 connection(s)
I20260927 00:50:25.776985 16672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:50:25.777583 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 16672
I20260927 00:50:25.787468 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:25.778360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33266 (local address 127.0.98.62:34319)
0927 00:50:25.778548 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 00:50:25.778553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:50:25.779002 (+ 449us) server_negotiation.cc:407] Connection header received
0927 00:50:25.779089 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:50:25.779093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:50:25.779143 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:50:25.779233 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:50:25.780063 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.780686 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.781942 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.782192 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.782276 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:25.783939 (+ 1663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:50:25.783943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:50:25.783946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:50:25.783976 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:50:25.785703 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:50:25.786158 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:50:25.786162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:50:25.786164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:50:25.786261 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:50:25.786620 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:50:25.786624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:50:25.786626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:50:25.786812 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:50:25.787052 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 00:50:25.787174 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 00:50:25.787295 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
Sep 27 00:50:25 dist-test-slave-86rs krb5kdc[15761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:50:25.793449 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:25.778841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48285 (local address 127.0.98.62:34319)
0927 00:50:25.779112 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 00:50:25.779117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:50:25.780087 (+ 970us) server_negotiation.cc:407] Connection header received
0927 00:50:25.781735 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:50:25.781739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:50:25.781796 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:50:25.781880 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:50:25.783535 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.784211 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.786087 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.786334 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.786489 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:25.789670 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:50:25.789673 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:50:25.789675 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:50:25.789697 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:50:25.791199 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:50:25.791746 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:50:25.791750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:50:25.791752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:50:25.791835 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:50:25.792201 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:50:25.792206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:50:25.792208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:50:25.792445 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:50:25.792554 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 00:50:25.793115 (+ 561us) server_negotiation.cc:299] Negotiation successful
0927 00:50:25.793256 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":95,"threads_started":1}
I20260927 00:50:25.794315 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:25.779427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.3:48285)
0927 00:50:25.779929 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:25.780007 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0927 00:50:25.781502 (+ 1495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:50:25.782099 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:50:25.782106 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:50:25.782632 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:50:25.783355 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:25.783378 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.784348 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.784352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:25.785965 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:25.785975 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.786548 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.786553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:25.786742 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:25.787335 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:50:25.787353 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:50:25.789431 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:50:25.791329 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:50:25.791336 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:50:25.791354 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:50:25.791635 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:50:25.791936 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:50:25.791940 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:50:25.791942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:50:25.792100 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:50:25.792568 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:50:25.792577 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:50:25.792861 (+ 284us) client_negotiation.cc:769] Sending connection context
0927 00:50:25.793075 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 00:50:25.793432 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":119,"threads_started":1}
I20260927 00:50:25.795580 16801 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34319
I20260927 00:50:25.795882 16801 heartbeater.cc:464] Registering TS with master...
I20260927 00:50:25.796551 16801 heartbeater.cc:511] Master 127.0.98.62:34319 requested a full tablet report, sending...
I20260927 00:50:25.797657 16343 ts_manager.cc:194] Registered new tserver with Master: ba6f939d15304d43b4212ea876b03187 (127.0.98.3:33475)
I20260927 00:50:25.798246 16343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48285
I20260927 00:50:25.801386 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:50:25.805398 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:25.801982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33276 (local address 127.0.98.62:34319)
0927 00:50:25.802133 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 00:50:25.802140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:50:25.802357 (+ 217us) server_negotiation.cc:407] Connection header received
0927 00:50:25.802486 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:50:25.802488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:50:25.802586 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:50:25.802676 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:50:25.802836 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.803345 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.804204 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.804520 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.804609 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:25.804889 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:50:25.805028 (+ 139us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 00:50:25.805253 (+ 225us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33276: token expired
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 00:50:25.805485 16804 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33276: token expired
W20260927 00:50:25.805855 16268 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:34319)
I20260927 00:50:25.813463 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:25.806202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33280 (local address 127.0.98.62:34319)
0927 00:50:25.806312 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 00:50:25.806316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:50:25.806336 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:50:25.806438 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:50:25.806440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:50:25.806477 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:50:25.806535 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:50:25.807141 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.807639 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.808545 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.808745 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.808816 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:25.810255 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:50:25.810258 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:50:25.810260 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:50:25.810284 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:50:25.811753 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:50:25.812203 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:50:25.812206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:50:25.812207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:50:25.812381 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:50:25.812672 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:50:25.812675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:50:25.812676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:50:25.812860 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:50:25.812968 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:50:25.813191 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 00:50:25.813302 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:50:25.816383 16343 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:50:25.819788 16343 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:50:25.828152 16465 tablet_service.cc:1467] Processing CreateTablet for tablet ee3f0d9898fa42a78a37130fc2d37ff5 (DEFAULT_TABLE table=test-table [id=4d4e579492c545e4a469f73598882ac0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:50:25.828943 16465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee3f0d9898fa42a78a37130fc2d37ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:50:25.831671 16423 tablet_bootstrap.cc:492] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a: Bootstrap starting.
I20260927 00:50:25.832801 16423 tablet_bootstrap.cc:654] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a: Neither blocks nor log segments found. Creating new log.
I20260927 00:50:25.833890 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:25.827346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43741 (local address 127.0.0.1:51896)
0927 00:50:25.827774 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:25.827789 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:50:25.827886 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:50:25.828602 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:50:25.828606 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:50:25.828629 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:50:25.828758 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:25.828765 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.830375 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.830378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:25.831610 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:25.831623 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.833286 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.833292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:25.833415 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:25.833505 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 00:50:25.833602 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 00:50:25.833704 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":45,"thread_start_us":128,"threads_started":1}
I20260927 00:50:25.833983 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:25.827731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51896 (local address 127.0.98.2:43741)
0927 00:50:25.828134 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0927 00:50:25.828145 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:50:25.828164 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:50:25.828259 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:50:25.828265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:50:25.828452 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:50:25.828602 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:50:25.828902 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.830248 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.832120 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.833092 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.833263 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:25.833453 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 00:50:25.833661 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 00:50:25.833799 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":119,"threads_started":1}
I20260927 00:50:25.834328 16538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:25.827093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33475 (local address 127.0.0.1:46984)
0927 00:50:25.827217 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:25.827236 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:50:25.827360 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:50:25.828466 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:50:25.828470 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:50:25.828486 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:50:25.828632 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:25.828643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.830183 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.830189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:25.831610 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:25.831623 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.833423 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.833427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:25.833991 (+ 564us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:25.834023 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 00:50:25.834106 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 00:50:25.834215 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":491}
I20260927 00:50:25.835021 16423 tablet_bootstrap.cc:492] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a: No bootstrap required, opened a new log
I20260927 00:50:25.835122 16423 ts_tablet_manager.cc:1397] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:50:25.835600 16423 raft_consensus.cc:348] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 } } peers { permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 } } peers { permanent_uuid: "a52f4e455782487687acb63ecff91e1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260927 00:50:25.835726 16423 raft_consensus.cc:374] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:50:25.835767 16423 raft_consensus.cc:729] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a52f4e455782487687acb63ecff91e1a, State: Initialized, Role: FOLLOWER
I20260927 00:50:25.836203 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:25.827307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46984 (local address 127.0.98.3:33475)
0927 00:50:25.827892 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0927 00:50:25.827905 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:50:25.827932 (+ 27us) server_negotiation.cc:407] Connection header received
0927 00:50:25.828017 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:50:25.828023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:50:25.828283 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:50:25.828788 (+ 505us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:50:25.828966 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.830072 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.831996 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.833293 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.833485 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:25.833634 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 00:50:25.835943 (+ 2309us) server_negotiation.cc:299] Negotiation successful
0927 00:50:25.836062 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":166,"threads_started":1}
I20260927 00:50:25.836493 16423 consensus_queue.cc:260] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 } } peers { permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 } } peers { permanent_uuid: "a52f4e455782487687acb63ecff91e1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260927 00:50:25.836812 16423 ts_tablet_manager.cc:1428] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:50:25.837478 16603 tablet_service.cc:1467] Processing CreateTablet for tablet ee3f0d9898fa42a78a37130fc2d37ff5 (DEFAULT_TABLE table=test-table [id=4d4e579492c545e4a469f73598882ac0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:50:25.838474 16603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee3f0d9898fa42a78a37130fc2d37ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:50:25.839251 16735 tablet_service.cc:1467] Processing CreateTablet for tablet ee3f0d9898fa42a78a37130fc2d37ff5 (DEFAULT_TABLE table=test-table [id=4d4e579492c545e4a469f73598882ac0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:50:25.840436 16735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee3f0d9898fa42a78a37130fc2d37ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:50:25.845650 16814 tablet_bootstrap.cc:492] T ee3f0d9898fa42a78a37130fc2d37ff5 P 5dd062a9695e41c59a75ef425fb7deed: Bootstrap starting.
I20260927 00:50:25.847071 16815 tablet_bootstrap.cc:492] T ee3f0d9898fa42a78a37130fc2d37ff5 P ba6f939d15304d43b4212ea876b03187: Bootstrap starting.
I20260927 00:50:25.848515 16814 tablet_bootstrap.cc:654] T ee3f0d9898fa42a78a37130fc2d37ff5 P 5dd062a9695e41c59a75ef425fb7deed: Neither blocks nor log segments found. Creating new log.
I20260927 00:50:25.849434 16815 tablet_bootstrap.cc:654] T ee3f0d9898fa42a78a37130fc2d37ff5 P ba6f939d15304d43b4212ea876b03187: Neither blocks nor log segments found. Creating new log.
I20260927 00:50:25.849424 16814 log.cc:824] T ee3f0d9898fa42a78a37130fc2d37ff5 P 5dd062a9695e41c59a75ef425fb7deed: Log is configured to *not* fsync() on all Append() calls
I20260927 00:50:25.850698 16815 log.cc:824] T ee3f0d9898fa42a78a37130fc2d37ff5 P ba6f939d15304d43b4212ea876b03187: Log is configured to *not* fsync() on all Append() calls
I20260927 00:50:25.851835 16814 tablet_bootstrap.cc:492] T ee3f0d9898fa42a78a37130fc2d37ff5 P 5dd062a9695e41c59a75ef425fb7deed: No bootstrap required, opened a new log
I20260927 00:50:25.852044 16814 ts_tablet_manager.cc:1397] T ee3f0d9898fa42a78a37130fc2d37ff5 P 5dd062a9695e41c59a75ef425fb7deed: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 00:50:25.852909 16815 tablet_bootstrap.cc:492] T ee3f0d9898fa42a78a37130fc2d37ff5 P ba6f939d15304d43b4212ea876b03187: No bootstrap required, opened a new log
I20260927 00:50:25.853106 16815 ts_tablet_manager.cc:1397] T ee3f0d9898fa42a78a37130fc2d37ff5 P ba6f939d15304d43b4212ea876b03187: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:50:25.856252 16814 raft_consensus.cc:348] T ee3f0d9898fa42a78a37130fc2d37ff5 P 5dd062a9695e41c59a75ef425fb7deed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 } } peers { permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 } } peers { permanent_uuid: "a52f4e455782487687acb63ecff91e1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260927 00:50:25.856658 16814 raft_consensus.cc:374] T ee3f0d9898fa42a78a37130fc2d37ff5 P 5dd062a9695e41c59a75ef425fb7deed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:50:25.856753 16814 raft_consensus.cc:729] T ee3f0d9898fa42a78a37130fc2d37ff5 P 5dd062a9695e41c59a75ef425fb7deed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd062a9695e41c59a75ef425fb7deed, State: Initialized, Role: FOLLOWER
I20260927 00:50:25.856971 16815 raft_consensus.cc:348] T ee3f0d9898fa42a78a37130fc2d37ff5 P ba6f939d15304d43b4212ea876b03187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 } } peers { permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 } } peers { permanent_uuid: "a52f4e455782487687acb63ecff91e1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260927 00:50:25.857246 16815 raft_consensus.cc:374] T ee3f0d9898fa42a78a37130fc2d37ff5 P ba6f939d15304d43b4212ea876b03187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:50:25.857313 16815 raft_consensus.cc:729] T ee3f0d9898fa42a78a37130fc2d37ff5 P ba6f939d15304d43b4212ea876b03187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba6f939d15304d43b4212ea876b03187, State: Initialized, Role: FOLLOWER
I20260927 00:50:25.857270 16814 consensus_queue.cc:260] T ee3f0d9898fa42a78a37130fc2d37ff5 P 5dd062a9695e41c59a75ef425fb7deed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 } } peers { permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 } } peers { permanent_uuid: "a52f4e455782487687acb63ecff91e1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260927 00:50:25.857811 16815 consensus_queue.cc:260] T ee3f0d9898fa42a78a37130fc2d37ff5 P ba6f939d15304d43b4212ea876b03187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 } } peers { permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 } } peers { permanent_uuid: "a52f4e455782487687acb63ecff91e1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260927 00:50:25.858458 16669 heartbeater.cc:503] Master 127.0.98.62:34319 was elected leader, sending a full tablet report...
I20260927 00:50:25.858510 16801 heartbeater.cc:503] Master 127.0.98.62:34319 was elected leader, sending a full tablet report...
I20260927 00:50:25.858868 16815 ts_tablet_manager.cc:1428] T ee3f0d9898fa42a78a37130fc2d37ff5 P ba6f939d15304d43b4212ea876b03187: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:50:25.858876 16814 ts_tablet_manager.cc:1428] T ee3f0d9898fa42a78a37130fc2d37ff5 P 5dd062a9695e41c59a75ef425fb7deed: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260927 00:50:25.875526 16532 tablet.cc:2369] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:50:25.881357 16537 raft_consensus.cc:482] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:50:25.881599 16537 raft_consensus.cc:504] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 } } peers { permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 } } peers { permanent_uuid: "a52f4e455782487687acb63ecff91e1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260927 00:50:25.882763 16537 leader_election.cc:288] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba6f939d15304d43b4212ea876b03187 (127.0.98.3:33475), 5dd062a9695e41c59a75ef425fb7deed (127.0.98.2:43741)
I20260927 00:50:25.886897 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:25.883089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33475 (local address 127.0.98.1:56805)
0927 00:50:25.883224 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:25.883260 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 00:50:25.883441 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:50:25.883802 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:50:25.883805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:50:25.883830 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:50:25.883963 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:25.883969 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.884783 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.884788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:25.885640 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:25.885650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.886357 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.886360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:25.886466 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:25.886543 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 00:50:25.886655 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 00:50:25.886746 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 00:50:25.886897 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:25.883193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56805 (local address 127.0.98.3:33475)
0927 00:50:25.883339 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 00:50:25.883343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:50:25.883358 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:50:25.883611 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:50:25.883614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:50:25.883674 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:50:25.883792 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:50:25.884080 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.884669 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.885777 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.886218 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.886349 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:25.886415 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 00:50:25.886697 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 00:50:25.886773 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 00:50:25.887051 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:25.883568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48705 (local address 127.0.98.2:43741)
0927 00:50:25.883708 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 00:50:25.883713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:50:25.883913 (+ 200us) server_negotiation.cc:407] Connection header received
0927 00:50:25.884056 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:50:25.884059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:50:25.884112 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:50:25.884196 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:50:25.884482 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.884978 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.886004 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.886428 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.886552 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:25.886613 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 00:50:25.886862 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 00:50:25.886940 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:50:25.887064 16820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:25.883409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43741 (local address 127.0.98.1:48705)
0927 00:50:25.883824 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:25.883834 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 00:50:25.883950 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:50:25.884204 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:50:25.884207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:50:25.884244 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:50:25.884375 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:25.884380 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.885086 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.885091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:25.885880 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:25.885889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.886562 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.886566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:25.886726 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:25.886747 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:50:25.886846 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 00:50:25.886922 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":127,"threads_started":1}
I20260927 00:50:25.887712 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee3f0d9898fa42a78a37130fc2d37ff5" candidate_uuid: "a52f4e455782487687acb63ecff91e1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd062a9695e41c59a75ef425fb7deed" is_pre_election: true
I20260927 00:50:25.887712 16755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee3f0d9898fa42a78a37130fc2d37ff5" candidate_uuid: "a52f4e455782487687acb63ecff91e1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba6f939d15304d43b4212ea876b03187" is_pre_election: true
I20260927 00:50:25.888090 16623 raft_consensus.cc:2454] T ee3f0d9898fa42a78a37130fc2d37ff5 P 5dd062a9695e41c59a75ef425fb7deed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a52f4e455782487687acb63ecff91e1a in term 0.
I20260927 00:50:25.888090 16755 raft_consensus.cc:2454] T ee3f0d9898fa42a78a37130fc2d37ff5 P ba6f939d15304d43b4212ea876b03187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a52f4e455782487687acb63ecff91e1a in term 0.
I20260927 00:50:25.888630 16418 leader_election.cc:302] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd062a9695e41c59a75ef425fb7deed, a52f4e455782487687acb63ecff91e1a; no voters:
I20260927 00:50:25.888931 16537 raft_consensus.cc:2781] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:50:25.889026 16537 raft_consensus.cc:482] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:50:25.889071 16537 raft_consensus.cc:3037] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:50:25.890352 16537 raft_consensus.cc:504] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 } } peers { permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 } } peers { permanent_uuid: "a52f4e455782487687acb63ecff91e1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260927 00:50:25.890707 16537 leader_election.cc:288] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [CANDIDATE]: Term 1 election: Requested vote from peers ba6f939d15304d43b4212ea876b03187 (127.0.98.3:33475), 5dd062a9695e41c59a75ef425fb7deed (127.0.98.2:43741)
I20260927 00:50:25.891129 16755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee3f0d9898fa42a78a37130fc2d37ff5" candidate_uuid: "a52f4e455782487687acb63ecff91e1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba6f939d15304d43b4212ea876b03187"
I20260927 00:50:25.891151 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee3f0d9898fa42a78a37130fc2d37ff5" candidate_uuid: "a52f4e455782487687acb63ecff91e1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd062a9695e41c59a75ef425fb7deed"
I20260927 00:50:25.891283 16755 raft_consensus.cc:3037] T ee3f0d9898fa42a78a37130fc2d37ff5 P ba6f939d15304d43b4212ea876b03187 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:50:25.891283 16623 raft_consensus.cc:3037] T ee3f0d9898fa42a78a37130fc2d37ff5 P 5dd062a9695e41c59a75ef425fb7deed [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:50:25.892681 16755 raft_consensus.cc:2454] T ee3f0d9898fa42a78a37130fc2d37ff5 P ba6f939d15304d43b4212ea876b03187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a52f4e455782487687acb63ecff91e1a in term 1.
I20260927 00:50:25.892680 16623 raft_consensus.cc:2454] T ee3f0d9898fa42a78a37130fc2d37ff5 P 5dd062a9695e41c59a75ef425fb7deed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a52f4e455782487687acb63ecff91e1a in term 1.
I20260927 00:50:25.893085 16418 leader_election.cc:302] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a52f4e455782487687acb63ecff91e1a, ba6f939d15304d43b4212ea876b03187; no voters:
I20260927 00:50:25.893272 16537 raft_consensus.cc:2781] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:50:25.893507 16537 raft_consensus.cc:686] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [term 1 LEADER]: Becoming Leader. State: Replica: a52f4e455782487687acb63ecff91e1a, State: Running, Role: LEADER
I20260927 00:50:25.893829 16537 consensus_queue.cc:237] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 } } peers { permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 } } peers { permanent_uuid: "a52f4e455782487687acb63ecff91e1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260927 00:50:25.897356 16343 catalog_manager.cc:5495] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a reported cstate change: term changed from 0 to 1, leader changed from <none> to a52f4e455782487687acb63ecff91e1a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a52f4e455782487687acb63ecff91e1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a52f4e455782487687acb63ecff91e1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } health_report { overall_health: HEALTHY } } }
I20260927 00:50:25.914073 16539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:25.910500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43824 (local address 127.0.98.1:40125)
0927 00:50:25.910637 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:50:25.910640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:50:25.910655 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:50:25.910732 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:50:25.910734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:50:25.910787 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:50:25.910879 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:50:25.911090 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.911652 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.912619 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:25.912877 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:25.912968 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:25.913196 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:50:25.913672 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:50:25.913762 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 00:50:25.913820 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 00:50:25.913880 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:50:25.918406 16623 raft_consensus.cc:1261] T ee3f0d9898fa42a78a37130fc2d37ff5 P 5dd062a9695e41c59a75ef425fb7deed [term 1 FOLLOWER]: Refusing update from remote peer a52f4e455782487687acb63ecff91e1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:50:25.918394 16755 raft_consensus.cc:1261] T ee3f0d9898fa42a78a37130fc2d37ff5 P ba6f939d15304d43b4212ea876b03187 [term 1 FOLLOWER]: Refusing update from remote peer a52f4e455782487687acb63ecff91e1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:50:25.919129 16537 consensus_queue.cc:1035] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:50:25.919340 16821 consensus_queue.cc:1035] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:50:25.930173 16824 mvcc.cc:205] Tried to move back new op lower bound from 7333766045353398272 to 7333766045262958592. Current Snapshot: MvccSnapshot[applied={T|T < 7333766045353398272 or (T in {7333766045353398272})}]
I20260927 00:50:25.937634 16343 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:50:25.937809 16343 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:50:25.940443 16343 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1: Sending DeleteTablet for 3 replicas of tablet ee3f0d9898fa42a78a37130fc2d37ff5
I20260927 00:50:25.941247 16465 tablet_service.cc:1514] Processing DeleteTablet for tablet ee3f0d9898fa42a78a37130fc2d37ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:50:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43816
I20260927 00:50:25.941591 16603 tablet_service.cc:1514] Processing DeleteTablet for tablet ee3f0d9898fa42a78a37130fc2d37ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:50:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51896
I20260927 00:50:25.941583 16735 tablet_service.cc:1514] Processing DeleteTablet for tablet ee3f0d9898fa42a78a37130fc2d37ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:50:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46984
I20260927 00:50:25.941733 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 16381
I20260927 00:50:25.942538 16835 tablet_replica.cc:330] T ee3f0d9898fa42a78a37130fc2d37ff5 P 5dd062a9695e41c59a75ef425fb7deed: stopping tablet replica
I20260927 00:50:25.942943 16835 raft_consensus.cc:2229] T ee3f0d9898fa42a78a37130fc2d37ff5 P 5dd062a9695e41c59a75ef425fb7deed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:50:25.943324 16835 raft_consensus.cc:2258] T ee3f0d9898fa42a78a37130fc2d37ff5 P 5dd062a9695e41c59a75ef425fb7deed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:50:25.944464 16834 tablet_replica.cc:330] T ee3f0d9898fa42a78a37130fc2d37ff5 P ba6f939d15304d43b4212ea876b03187: stopping tablet replica
I20260927 00:50:25.944789 16834 raft_consensus.cc:2229] T ee3f0d9898fa42a78a37130fc2d37ff5 P ba6f939d15304d43b4212ea876b03187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:50:25.945084 16835 ts_tablet_manager.cc:1905] T ee3f0d9898fa42a78a37130fc2d37ff5 P 5dd062a9695e41c59a75ef425fb7deed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:50:25.945119 16834 raft_consensus.cc:2258] T ee3f0d9898fa42a78a37130fc2d37ff5 P ba6f939d15304d43b4212ea876b03187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:50:25.947463 16834 ts_tablet_manager.cc:1905] T ee3f0d9898fa42a78a37130fc2d37ff5 P ba6f939d15304d43b4212ea876b03187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:50:25.948736 16835 ts_tablet_manager.cc:1918] T ee3f0d9898fa42a78a37130fc2d37ff5 P 5dd062a9695e41c59a75ef425fb7deed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:50:25.948858 16835 log.cc:1196] T ee3f0d9898fa42a78a37130fc2d37ff5 P 5dd062a9695e41c59a75ef425fb7deed: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/wal/wals/ee3f0d9898fa42a78a37130fc2d37ff5
I20260927 00:50:25.949290 16835 ts_tablet_manager.cc:1939] T ee3f0d9898fa42a78a37130fc2d37ff5 P 5dd062a9695e41c59a75ef425fb7deed: Deleting consensus metadata
I20260927 00:50:25.950001 16834 ts_tablet_manager.cc:1918] T ee3f0d9898fa42a78a37130fc2d37ff5 P ba6f939d15304d43b4212ea876b03187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:50:25.950119 16834 log.cc:1196] T ee3f0d9898fa42a78a37130fc2d37ff5 P ba6f939d15304d43b4212ea876b03187: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/wal/wals/ee3f0d9898fa42a78a37130fc2d37ff5
I20260927 00:50:25.950443 16834 ts_tablet_manager.cc:1939] T ee3f0d9898fa42a78a37130fc2d37ff5 P ba6f939d15304d43b4212ea876b03187: Deleting consensus metadata
I20260927 00:50:25.950743 16328 catalog_manager.cc:4841] TS 5dd062a9695e41c59a75ef425fb7deed (127.0.98.2:43741): tablet ee3f0d9898fa42a78a37130fc2d37ff5 (table test-table [id=4d4e579492c545e4a469f73598882ac0]) successfully deleted
I20260927 00:50:25.950960 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 16535
W20260927 00:50:25.950942 16327 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40125 (error 108)
I20260927 00:50:25.952561 16538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:25.952270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40125 (local address 127.0.0.1:43834)
0927 00:50:25.952399 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:25.952478 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 00:50:25.952725 16327 catalog_manager.cc:4584] TS a52f4e455782487687acb63ecff91e1a (127.0.98.1:40125): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee3f0d9898fa42a78a37130fc2d37ff5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40125: connect: Connection refused (error 111)
I20260927 00:50:25.954856 16328 catalog_manager.cc:4841] TS ba6f939d15304d43b4212ea876b03187 (127.0.98.3:33475): tablet ee3f0d9898fa42a78a37130fc2d37ff5 (table test-table [id=4d4e579492c545e4a469f73598882ac0]) successfully deleted
I20260927 00:50:25.957870 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 16672
I20260927 00:50:25.964397 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 16310
I20260927 00:50:45.971237 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34319
--webserver_interface=127.0.98.62
--webserver_port=41639
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:50:46.088999 16837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:50:46.089295 16837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:50:46.089345 16837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:50:46.093637 16837 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 00:50:46.093705 16837 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 00:50:46.093722 16837 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 00:50:46.093736 16837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:50:46.093756 16837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:50:46.093798 16837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:50:46.093832 16837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:50:46.093858 16837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:50:46.099557 16837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:50:46.100819 16837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:50:46.101894 16837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:50:46.111249 16837 server_base.cc:1034] running on GCE node
W20260927 00:50:46.111294 16842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:50:46.111370 16845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:50:46.111249 16843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:50:46.112160 16837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:50:46.113399 16837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:50:46.114598 16837 hybrid_clock.cc:648] HybridClock initialized: now 1790470246114581 us; error 35 us; skew 500 ppm
Sep 27 00:50:46 dist-test-slave-86rs krb5kdc[15761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:50:46.117476 16837 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:50:46.118451 16837 webserver.cc:460] Webserver started at http://127.0.98.62:41639/ using document root <none> and password file <none>
I20260927 00:50:46.119004 16837 fs_manager.cc:359] Metadata directory not provided
I20260927 00:50:46.119065 16837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:50:46.123159 16837 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:50:46.125537 16852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:50:46.126871 16837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:50:46.127012 16837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "f491678175874e80b10a2e572a51ccd1"
format_stamp: "Formatted at 2026-09-27 00:50:03 on dist-test-slave-86rs"
server_key: "76045a0732c9276322172d0aa3f8cc2e"
server_key_iv: "b8cc60c006346ca8afe1a7ff8ac77b7b"
server_key_version: "encryptionkey@0"
I20260927 00:50:46.127604 16837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:50:46.142211 16837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:50:46.145648 16837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:50:46.145872 16837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:50:46.153523 16837 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34319
I20260927 00:50:46.153841 16904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34319 every 8 connection(s)
I20260927 00:50:46.154735 16837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:50:46.158524 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 16837
I20260927 00:50:46.159067 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40125
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34319
--webserver_port=45759
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:50:46.159974 16905 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:50:46.162106 16905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1: Bootstrap starting.
I20260927 00:50:46.180707 16905 log.cc:824] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1: Log is configured to *not* fsync() on all Append() calls
I20260927 00:50:46.192618 16905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:50:46.193081 16905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1: Bootstrap complete.
I20260927 00:50:46.197475 16905 raft_consensus.cc:348] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f491678175874e80b10a2e572a51ccd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } }
I20260927 00:50:46.197902 16905 raft_consensus.cc:729] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f491678175874e80b10a2e572a51ccd1, State: Initialized, Role: FOLLOWER
I20260927 00:50:46.198498 16905 consensus_queue.cc:260] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f491678175874e80b10a2e572a51ccd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } }
I20260927 00:50:46.198629 16905 raft_consensus.cc:388] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:50:46.198717 16905 raft_consensus.cc:482] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:50:46.198841 16905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [term 2 FOLLOWER]: Advancing to term 3
I20260927 00:50:46.200992 16905 raft_consensus.cc:504] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f491678175874e80b10a2e572a51ccd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } }
I20260927 00:50:46.201359 16905 leader_election.cc:302] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f491678175874e80b10a2e572a51ccd1; no voters:
I20260927 00:50:46.201719 16905 leader_election.cc:288] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 00:50:46.201817 16909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 00:50:46.202029 16909 raft_consensus.cc:686] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [term 3 LEADER]: Becoming Leader. State: Replica: f491678175874e80b10a2e572a51ccd1, State: Running, Role: LEADER
I20260927 00:50:46.202419 16909 consensus_queue.cc:237] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f491678175874e80b10a2e572a51ccd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } }
I20260927 00:50:46.203087 16905 sys_catalog.cc:564] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:50:46.204123 16911 sys_catalog.cc:455] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f491678175874e80b10a2e572a51ccd1. Latest consensus state: current_term: 3 leader_uuid: "f491678175874e80b10a2e572a51ccd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f491678175874e80b10a2e572a51ccd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } } }
I20260927 00:50:46.204313 16911 sys_catalog.cc:458] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [sys.catalog]: This master's current role is: LEADER
I20260927 00:50:46.204639 16910 sys_catalog.cc:455] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f491678175874e80b10a2e572a51ccd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f491678175874e80b10a2e572a51ccd1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } } }
I20260927 00:50:46.204744 16910 sys_catalog.cc:458] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1 [sys.catalog]: This master's current role is: LEADER
I20260927 00:50:46.204756 16918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:50:46.210407 16918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:50:46.211231 16918 catalog_manager.cc:1259] Loaded cluster ID: f18e887dadde4191a8a38af6fa42730e
I20260927 00:50:46.211319 16918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:50:46.213310 16918 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:50:46.214059 16918 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1: Loaded TSK: 0
I20260927 00:50:46.214151 16918 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f491678175874e80b10a2e572a51ccd1: Loaded TSK: 1
F20260927 00:50:46.214794 16918 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9b1a3f39ad google::LogMessage::Fail() at ??:0
@ 0x7f9b1a3f688a google::LogMessage::SendToLog() at ??:0
@ 0x7f9b1a3f34de google::LogMessage::Flush() at ??:0
@ 0x7f9b1a3f4a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9b1d0b718d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f9b1d0aff0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f9b24d587bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f9b24d3299c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f9b24d33178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f9b24d74df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f9b24d9382b std::function<>::operator()() at ??:0
@ 0x7f9b24d32b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f9b24d33f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9b24d30579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f9b24d74640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f9b1c6453d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9b1c645c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f9b1c6473f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f9b1c636512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9b1dc7a6db start_thread at ??:0
@ 0x7f9b18ef161f clone at ??:0
W20260927 00:50:46.320650 16907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:50:46.320951 16907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:50:46.320998 16907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:50:46.325278 16907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:50:46.325351 16907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:50:46.325443 16907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:50:46.331717 16907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40125
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45759
--tserver_master_addrs=127.0.98.62:34319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:50:46.332993 16907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:50:46.334129 16907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:50:46.344745 16953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:50:46.344817 16952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:50:46.344755 16955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:50:46.345391 16907 server_base.cc:1034] running on GCE node
I20260927 00:50:46.345914 16907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:50:46.346773 16907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:50:46.348011 16907 hybrid_clock.cc:648] HybridClock initialized: now 1790470246347993 us; error 28 us; skew 500 ppm
Sep 27 00:50:46 dist-test-slave-86rs krb5kdc[15761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:50:46.351604 16907 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:50:46.352697 16907 webserver.cc:460] Webserver started at http://127.0.98.1:45759/ using document root <none> and password file <none>
I20260927 00:50:46.353278 16907 fs_manager.cc:359] Metadata directory not provided
I20260927 00:50:46.353341 16907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:50:46.357726 16907 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:50:46.361873 16962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:50:46.363420 16907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:50:46.363588 16907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "a52f4e455782487687acb63ecff91e1a"
format_stamp: "Formatted at 2026-09-27 00:50:04 on dist-test-slave-86rs"
server_key: "a6c3acbb676d08ca37fbc3b5e6ae7f57"
server_key_iv: "4303f9cd97d5b3901cac3b9b76c016cd"
server_key_version: "encryptionkey@0"
I20260927 00:50:46.364195 16907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:50:46.377209 16907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:50:46.381058 16907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:50:46.381285 16907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:50:46.381942 16907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:50:46.383124 16969 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 00:50:46.389178 16907 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 00:50:46.389312 16907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 00:50:46.389417 16907 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 00:50:46.391273 16907 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 00:50:46.391355 16907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 00:50:46.391680 16969 tablet_bootstrap.cc:492] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a: Bootstrap starting.
I20260927 00:50:46.403127 16907 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40125
I20260927 00:50:46.403264 17076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40125 every 8 connection(s)
I20260927 00:50:46.404579 16907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:50:46.407238 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 16907
I20260927 00:50:46.407752 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43741
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34319
--webserver_port=34355
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:50:46.414047 16969 log.cc:824] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a: Log is configured to *not* fsync() on all Append() calls
I20260927 00:50:46.424801 16969 tablet_bootstrap.cc:492] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:50:46.425274 16969 tablet_bootstrap.cc:492] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a: Bootstrap complete.
I20260927 00:50:46.426740 16969 ts_tablet_manager.cc:1397] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a: Time spent bootstrapping tablet: real 0.035s user 0.025s sys 0.007s
I20260927 00:50:46.430284 16969 raft_consensus.cc:348] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 } } peers { permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 } } peers { permanent_uuid: "a52f4e455782487687acb63ecff91e1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260927 00:50:46.430859 16969 raft_consensus.cc:729] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a52f4e455782487687acb63ecff91e1a, State: Initialized, Role: FOLLOWER
I20260927 00:50:46.431401 16969 consensus_queue.cc:260] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 } } peers { permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 } } peers { permanent_uuid: "a52f4e455782487687acb63ecff91e1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260927 00:50:46.432543 16969 ts_tablet_manager.cc:1428] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260927 00:50:46.452627 17079 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:50:46.407084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.1:33581)
0927 00:50:46.408675 (+ 1591us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:46.408754 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0927 00:50:46.409882 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:50:46.451775 (+ 41893us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1331,"thread_start_us":148,"threads_started":1}
W20260927 00:50:46.453394 17077 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34319 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34319: Client connection negotiation failed: client connection to 127.0.98.62:34319: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 00:50:46.454205 17079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:46.453950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.1:57565)
0927 00:50:46.454079 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:46.454145 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 00:50:46.454893 17079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:46.454682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.1:58389)
0927 00:50:46.454790 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:46.454845 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 00:50:46.455070 17077 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:50:46.535367 17080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:50:46.535725 17080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:50:46.535810 17080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:50:46.540151 17080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:50:46.540270 17080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:50:46.540390 17080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:50:46.546185 17080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34355
--tserver_master_addrs=127.0.98.62:34319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:50:46.547391 17080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:50:46.548588 17080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:50:46.559316 17088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:50:46.559459 17080 server_base.cc:1034] running on GCE node
W20260927 00:50:46.559512 17090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:50:46.559612 17087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:50:46.560039 17080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:50:46.560773 17080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:50:46.561991 17080 hybrid_clock.cc:648] HybridClock initialized: now 1790470246561976 us; error 26 us; skew 500 ppm
Sep 27 00:50:46 dist-test-slave-86rs krb5kdc[15761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:50:46.565026 17080 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:50:46.565958 17080 webserver.cc:460] Webserver started at http://127.0.98.2:34355/ using document root <none> and password file <none>
I20260927 00:50:46.566524 17080 fs_manager.cc:359] Metadata directory not provided
I20260927 00:50:46.566584 17080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:50:46.570983 17080 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:50:46.573364 17097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:50:46.574784 17080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:50:46.574925 17080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "5dd062a9695e41c59a75ef425fb7deed"
format_stamp: "Formatted at 2026-09-27 00:50:04 on dist-test-slave-86rs"
server_key: "e54e64010f47e010a8da134e7fe16a4c"
server_key_iv: "67e542ec782de4bf0e996c80619b0aa5"
server_key_version: "encryptionkey@0"
I20260927 00:50:46.575517 17080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:50:46.589448 17080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:50:46.592877 17080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:50:46.593094 17080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:50:46.593750 17080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:50:46.594641 17080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:50:46.594689 17080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:50:46.594754 17080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:50:46.594786 17080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:50:46.603844 17080 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43741
I20260927 00:50:46.603873 17210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43741 every 8 connection(s)
I20260927 00:50:46.605064 17080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:50:46.607620 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 17080
I20260927 00:50:46.608112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33475
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34319
--webserver_port=38461
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:50:46.609311 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:46.607301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.2:33353)
0927 00:50:46.607831 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:46.607983 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":136,"threads_started":1}
W20260927 00:50:46.609963 17211 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34319 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34319: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
I20260927 00:50:46.610693 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:46.610450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.2:34157)
0927 00:50:46.610571 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:46.610628 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 00:50:46.611392 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:46.611193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.2:40655)
0927 00:50:46.611297 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:46.611338 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 00:50:46.611618 17211 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:50:46.724689 17214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:50:46.724994 17214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:50:46.725087 17214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:50:46.729395 17214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:50:46.729475 17214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:50:46.729573 17214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:50:46.735159 17214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33475
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38461
--tserver_master_addrs=127.0.98.62:34319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:50:46.736447 17214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:50:46.737542 17214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:50:46.747958 17214 server_base.cc:1034] running on GCE node
W20260927 00:50:46.747913 17220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:50:46.747888 17219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:50:46.747888 17222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:50:46.748660 17214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:50:46.749423 17214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:50:46.750640 17214 hybrid_clock.cc:648] HybridClock initialized: now 1790470246750620 us; error 40 us; skew 500 ppm
Sep 27 00:50:46 dist-test-slave-86rs krb5kdc[15761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:50:46.753784 17214 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:50:46.754889 17214 webserver.cc:460] Webserver started at http://127.0.98.3:38461/ using document root <none> and password file <none>
I20260927 00:50:46.755507 17214 fs_manager.cc:359] Metadata directory not provided
I20260927 00:50:46.755599 17214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:50:46.759704 17214 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:50:46.762123 17229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:50:46.763403 17214 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:50:46.763545 17214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "ba6f939d15304d43b4212ea876b03187"
format_stamp: "Formatted at 2026-09-27 00:50:04 on dist-test-slave-86rs"
server_key: "858c05533e410e04de2055482b692abf"
server_key_iv: "37b66fd31c71c957e6bee4e029eb8982"
server_key_version: "encryptionkey@0"
I20260927 00:50:46.764148 17214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:50:46.790071 17214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:50:46.793315 17214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:50:46.793541 17214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:50:46.794191 17214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:50:46.795091 17214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:50:46.795171 17214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:50:46.795243 17214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:50:46.795289 17214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:50:46.804870 17214 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33475
I20260927 00:50:46.804888 17342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33475 every 8 connection(s)
I20260927 00:50:46.806003 17214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:50:46.810077 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:46.808300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.3:41855)
0927 00:50:46.808768 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:46.808886 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":117,"threads_started":1}
W20260927 00:50:46.810607 17343 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34319 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34319: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
I20260927 00:50:46.811334 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:46.811068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.3:56825)
0927 00:50:46.811193 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:46.811243 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 00:50:46.812023 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:46.811835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.3:60795)
0927 00:50:46.811929 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:46.811971 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 00:50:46.812213 17343 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 00:50:46.815927 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 17214
I20260927 00:50:47.456189 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:47.455743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.1:40087)
0927 00:50:47.456007 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:47.456075 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":81,"threads_started":1}
I20260927 00:50:47.598433 17348 raft_consensus.cc:482] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:50:47.598635 17348 raft_consensus.cc:504] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 } } peers { permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 } } peers { permanent_uuid: "a52f4e455782487687acb63ecff91e1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260927 00:50:47.599749 17348 leader_election.cc:288] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ba6f939d15304d43b4212ea876b03187 (127.0.98.3:33475), 5dd062a9695e41c59a75ef425fb7deed (127.0.98.2:43741)
Sep 27 00:50:47 dist-test-slave-86rs krb5kdc[15761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 27 00:50:47 dist-test-slave-86rs krb5kdc[15761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260927 00:50:47.612824 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:47.612292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.2:60441)
0927 00:50:47.612603 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:47.612675 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":81,"threads_started":1}
I20260927 00:50:47.616430 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:47.600275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45221 (local address 127.0.98.3:33475)
0927 00:50:47.600729 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0927 00:50:47.600743 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:50:47.600781 (+ 38us) server_negotiation.cc:407] Connection header received
0927 00:50:47.601458 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:50:47.601464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:50:47.601864 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:50:47.602166 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:50:47.603767 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:47.604861 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:47.606519 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:47.607210 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:47.607813 (+ 603us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:47.610604 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:50:47.610609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:50:47.610623 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:50:47.610659 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:50:47.613045 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:50:47.613878 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:50:47.613887 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:50:47.613896 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:50:47.614080 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:50:47.614611 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:50:47.614615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:50:47.614616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:50:47.615145 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:50:47.615257 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 00:50:47.615887 (+ 630us) server_negotiation.cc:299] Negotiation successful
0927 00:50:47.616164 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":99,"threads_started":1}
I20260927 00:50:47.616434 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:47.600642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39487 (local address 127.0.98.2:43741)
0927 00:50:47.600934 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0927 00:50:47.600948 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:50:47.601162 (+ 214us) server_negotiation.cc:407] Connection header received
0927 00:50:47.601901 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:50:47.601906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:50:47.602339 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:50:47.602952 (+ 613us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:50:47.603743 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:47.605153 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:47.606552 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:50:47.607100 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:50:47.607271 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:47.610667 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:50:47.610671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:50:47.610683 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:50:47.610718 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:50:47.613697 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:50:47.614249 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:50:47.614255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:50:47.614261 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:50:47.614413 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:50:47.614782 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:50:47.614785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:50:47.614787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:50:47.615251 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:50:47.615393 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 00:50:47.615892 (+ 499us) server_negotiation.cc:299] Negotiation successful
0927 00:50:47.616171 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":105,"threads_started":1}
I20260927 00:50:47.616473 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:47.600452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43741 (local address 127.0.98.1:39487)
0927 00:50:47.601043 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:47.601055 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:50:47.601191 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:50:47.602703 (+ 1512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:50:47.602706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:50:47.602970 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:50:47.603533 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:47.603551 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:47.605289 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:47.605292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:47.606384 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:47.606392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:47.607241 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:47.607245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:47.607434 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:47.608194 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:50:47.608206 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:50:47.610067 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:50:47.613842 (+ 3775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:50:47.613846 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:50:47.613848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:50:47.614109 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:50:47.614536 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:50:47.614538 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:50:47.614539 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:50:47.614660 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:50:47.615374 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:50:47.615377 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:50:47.615567 (+ 190us) client_negotiation.cc:769] Sending connection context
0927 00:50:47.615769 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 00:50:47.616158 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":179,"spinlock_wait_cycles":29440,"thread_start_us":80,"threads_started":1}
I20260927 00:50:47.617018 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:47.600133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33475 (local address 127.0.98.1:45221)
0927 00:50:47.600282 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:47.600300 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:50:47.600415 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:50:47.602242 (+ 1827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:50:47.602249 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:50:47.602932 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:50:47.603536 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:47.603554 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:47.605016 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:47.605031 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:47.606380 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:50:47.606392 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:50:47.607372 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:50:47.607376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:50:47.607477 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:50:47.608148 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:50:47.608173 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:50:47.610387 (+ 2214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:50:47.613269 (+ 2882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:50:47.613325 (+ 56us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:50:47.613345 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:50:47.613760 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:50:47.614235 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:50:47.614238 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:50:47.614240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:50:47.614503 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:50:47.615287 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:50:47.615297 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:50:47.615568 (+ 271us) client_negotiation.cc:769] Sending connection context
0927 00:50:47.615769 (+ 201us) client_negotiation.cc:240] Negotiation successful
0927 00:50:47.616297 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":198,"spinlock_wait_cycles":42880}
W20260927 00:50:47.619563 16964 leader_election.cc:341] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dd062a9695e41c59a75ef425fb7deed (127.0.98.2:43741): Not found: Tablet not found: ee3f0d9898fa42a78a37130fc2d37ff5
W20260927 00:50:47.619798 16964 leader_election.cc:341] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ba6f939d15304d43b4212ea876b03187 (127.0.98.3:33475): Not found: Tablet not found: ee3f0d9898fa42a78a37130fc2d37ff5
I20260927 00:50:47.619866 16964 leader_election.cc:302] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a52f4e455782487687acb63ecff91e1a; no voters: 5dd062a9695e41c59a75ef425fb7deed, ba6f939d15304d43b4212ea876b03187
I20260927 00:50:47.620169 17348 raft_consensus.cc:2726] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:50:47.813349 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:47.812836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.3:47105)
0927 00:50:47.813144 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:47.813224 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":140,"threads_started":1}
I20260927 00:50:48.457564 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:48.457127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.1:54675)
0927 00:50:48.457405 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:48.457465 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":76,"threads_started":1}
I20260927 00:50:48.614132 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:48.613643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.2:35011)
0927 00:50:48.613936 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:48.613997 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":79,"threads_started":1}
I20260927 00:50:48.814755 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:48.814222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.3:49521)
0927 00:50:48.814548 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:48.814614 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":114,"threads_started":1}
I20260927 00:50:49.394541 17357 raft_consensus.cc:482] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:50:49.394766 17357 raft_consensus.cc:504] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 } } peers { permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 } } peers { permanent_uuid: "a52f4e455782487687acb63ecff91e1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260927 00:50:49.395290 17357 leader_election.cc:288] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ba6f939d15304d43b4212ea876b03187 (127.0.98.3:33475), 5dd062a9695e41c59a75ef425fb7deed (127.0.98.2:43741)
W20260927 00:50:49.396510 16964 leader_election.cc:341] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ba6f939d15304d43b4212ea876b03187 (127.0.98.3:33475): Not found: Tablet not found: ee3f0d9898fa42a78a37130fc2d37ff5
W20260927 00:50:49.396675 16964 leader_election.cc:341] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dd062a9695e41c59a75ef425fb7deed (127.0.98.2:43741): Not found: Tablet not found: ee3f0d9898fa42a78a37130fc2d37ff5
I20260927 00:50:49.396736 16964 leader_election.cc:302] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a52f4e455782487687acb63ecff91e1a; no voters: 5dd062a9695e41c59a75ef425fb7deed, ba6f939d15304d43b4212ea876b03187
I20260927 00:50:49.396973 17357 raft_consensus.cc:2726] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:50:49.459249 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:49.458694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.1:36667)
0927 00:50:49.459042 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:49.459121 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":119,"threads_started":1}
I20260927 00:50:49.615751 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:49.615182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.2:45731)
0927 00:50:49.615498 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:49.615572 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":102,"threads_started":1}
I20260927 00:50:49.816303 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:49.815752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.3:45491)
0927 00:50:49.816091 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:49.816158 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":122,"threads_started":1}
I20260927 00:50:50.460769 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:50.460208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.1:49227)
0927 00:50:50.460594 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:50.460663 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":112,"threads_started":1}
I20260927 00:50:50.617344 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:50.616852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.2:55857)
0927 00:50:50.617165 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:50.617235 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":98,"threads_started":1}
I20260927 00:50:50.817763 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:50.817237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.3:35849)
0927 00:50:50.817567 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:50.817630 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":142,"threads_started":1}
I20260927 00:50:51.462389 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:51.461828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.1:52601)
0927 00:50:51.462222 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:51.462284 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":77,"threads_started":1}
I20260927 00:50:51.618762 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:51.618307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.2:39507)
0927 00:50:51.618594 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:51.618656 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":80,"threads_started":1}
I20260927 00:50:51.819089 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:51.818658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.3:38793)
0927 00:50:51.818932 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:51.818991 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":76,"threads_started":1}
I20260927 00:50:52.195312 17367 raft_consensus.cc:482] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:50:52.195467 17367 raft_consensus.cc:504] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 } } peers { permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 } } peers { permanent_uuid: "a52f4e455782487687acb63ecff91e1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260927 00:50:52.196039 17367 leader_election.cc:288] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ba6f939d15304d43b4212ea876b03187 (127.0.98.3:33475), 5dd062a9695e41c59a75ef425fb7deed (127.0.98.2:43741)
W20260927 00:50:52.197027 16964 leader_election.cc:341] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ba6f939d15304d43b4212ea876b03187 (127.0.98.3:33475): Not found: Tablet not found: ee3f0d9898fa42a78a37130fc2d37ff5
W20260927 00:50:52.197186 16964 leader_election.cc:341] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dd062a9695e41c59a75ef425fb7deed (127.0.98.2:43741): Not found: Tablet not found: ee3f0d9898fa42a78a37130fc2d37ff5
I20260927 00:50:52.197256 16964 leader_election.cc:302] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a52f4e455782487687acb63ecff91e1a; no voters: 5dd062a9695e41c59a75ef425fb7deed, ba6f939d15304d43b4212ea876b03187
I20260927 00:50:52.197496 17367 raft_consensus.cc:2726] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:50:52.463771 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:52.463337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.1:43557)
0927 00:50:52.463608 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:52.463672 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":75,"threads_started":1}
I20260927 00:50:52.620182 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:52.619694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.2:54445)
0927 00:50:52.619984 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:52.620059 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":95,"threads_started":1}
I20260927 00:50:52.820572 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:52.820009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.3:45603)
0927 00:50:52.820398 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:52.820472 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":80,"threads_started":1}
I20260927 00:50:53.465190 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:53.464736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.1:33301)
0927 00:50:53.465023 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:53.465091 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":80,"threads_started":1}
I20260927 00:50:53.621892 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:53.621145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.2:57685)
0927 00:50:53.621549 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:53.621622 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":80,"threads_started":1}
I20260927 00:50:53.822160 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:53.821530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.3:48565)
0927 00:50:53.821975 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:53.822052 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":97,"threads_started":1}
I20260927 00:50:54.466758 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:54.466223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.1:56565)
0927 00:50:54.466581 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:54.466649 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":119,"threads_started":1}
I20260927 00:50:54.623412 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:54.622865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.2:55935)
0927 00:50:54.623199 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:54.623270 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":103,"threads_started":1}
I20260927 00:50:54.823637 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:54.823164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.3:58697)
0927 00:50:54.823463 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:54.823537 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":86,"threads_started":1}
I20260927 00:50:55.468181 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:55.467726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.1:36587)
0927 00:50:55.468012 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:55.468079 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":78,"threads_started":1}
I20260927 00:50:55.625079 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:55.624598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.2:42837)
0927 00:50:55.624901 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:55.624967 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":84,"threads_started":1}
I20260927 00:50:55.825112 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:55.824568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.3:46649)
0927 00:50:55.824881 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:55.824997 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":90,"threads_started":1}
I20260927 00:50:56.371686 17380 raft_consensus.cc:482] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:50:56.371814 17380 raft_consensus.cc:504] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 } } peers { permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 } } peers { permanent_uuid: "a52f4e455782487687acb63ecff91e1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260927 00:50:56.372287 17380 leader_election.cc:288] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ba6f939d15304d43b4212ea876b03187 (127.0.98.3:33475), 5dd062a9695e41c59a75ef425fb7deed (127.0.98.2:43741)
W20260927 00:50:56.373215 16964 leader_election.cc:341] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dd062a9695e41c59a75ef425fb7deed (127.0.98.2:43741): Not found: Tablet not found: ee3f0d9898fa42a78a37130fc2d37ff5
W20260927 00:50:56.373356 16964 leader_election.cc:341] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ba6f939d15304d43b4212ea876b03187 (127.0.98.3:33475): Not found: Tablet not found: ee3f0d9898fa42a78a37130fc2d37ff5
I20260927 00:50:56.373440 16964 leader_election.cc:302] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a52f4e455782487687acb63ecff91e1a; no voters: 5dd062a9695e41c59a75ef425fb7deed, ba6f939d15304d43b4212ea876b03187
I20260927 00:50:56.373585 17380 raft_consensus.cc:2726] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:50:56.469535 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:56.469121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.1:49929)
0927 00:50:56.469379 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:56.469437 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":70,"threads_started":1}
I20260927 00:50:56.626495 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:56.626024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.2:53143)
0927 00:50:56.626310 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:56.626371 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":89,"threads_started":1}
I20260927 00:50:56.826371 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:56.825906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.3:50741)
0927 00:50:56.826203 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:56.826267 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":81,"threads_started":1}
I20260927 00:50:57.470774 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:57.470337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.1:52179)
0927 00:50:57.470607 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:57.470672 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":86,"threads_started":1}
I20260927 00:50:57.627912 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:57.627419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.2:34505)
0927 00:50:57.627707 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:57.627779 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":78,"threads_started":1}
I20260927 00:50:57.827759 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:57.827265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.3:52093)
0927 00:50:57.827573 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:57.827642 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":94,"threads_started":1}
I20260927 00:50:58.472049 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:58.471618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.1:35017)
0927 00:50:58.471875 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:58.471938 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":75,"threads_started":1}
I20260927 00:50:58.629304 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:58.628843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.2:58103)
0927 00:50:58.629133 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:58.629199 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":77,"threads_started":1}
I20260927 00:50:58.829205 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:58.828747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.3:44695)
0927 00:50:58.829041 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:58.829105 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":93,"threads_started":1}
I20260927 00:50:59.247792 17390 raft_consensus.cc:482] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:50:59.247929 17390 raft_consensus.cc:504] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f939d15304d43b4212ea876b03187" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33475 } } peers { permanent_uuid: "5dd062a9695e41c59a75ef425fb7deed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43741 } } peers { permanent_uuid: "a52f4e455782487687acb63ecff91e1a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260927 00:50:59.248487 17390 leader_election.cc:288] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ba6f939d15304d43b4212ea876b03187 (127.0.98.3:33475), 5dd062a9695e41c59a75ef425fb7deed (127.0.98.2:43741)
W20260927 00:50:59.249320 16964 leader_election.cc:341] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ba6f939d15304d43b4212ea876b03187 (127.0.98.3:33475): Not found: Tablet not found: ee3f0d9898fa42a78a37130fc2d37ff5
W20260927 00:50:59.249490 16964 leader_election.cc:341] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5dd062a9695e41c59a75ef425fb7deed (127.0.98.2:43741): Not found: Tablet not found: ee3f0d9898fa42a78a37130fc2d37ff5
I20260927 00:50:59.249557 16964 leader_election.cc:302] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a52f4e455782487687acb63ecff91e1a; no voters: 5dd062a9695e41c59a75ef425fb7deed, ba6f939d15304d43b4212ea876b03187
I20260927 00:50:59.249744 17390 raft_consensus.cc:2726] T ee3f0d9898fa42a78a37130fc2d37ff5 P a52f4e455782487687acb63ecff91e1a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:50:59.473264 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:59.472847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.1:59633)
0927 00:50:59.473105 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:59.473164 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":85,"threads_started":1}
I20260927 00:50:59.630667 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:59.630212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.2:59531)
0927 00:50:59.630499 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:59.630564 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":90,"threads_started":1}
I20260927 00:50:59.830485 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:50:59.830040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.3:58349)
0927 00:50:59.830312 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 00:50:59.830369 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":95,"threads_started":1}
I20260927 00:51:00.474870 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:00.474272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.1:55761)
0927 00:51:00.474676 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:00.474741 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":180,"threads_started":1}
I20260927 00:51:00.632211 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:00.631644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.2:59765)
0927 00:51:00.631990 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:00.632072 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":120,"threads_started":1}
I20260927 00:51:00.832207 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:00.831579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.3:41825)
0927 00:51:00.832005 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:00.832068 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":135,"threads_started":1}
I20260927 00:51:01.476692 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:01.476121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.1:60883)
0927 00:51:01.476505 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:01.476569 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":185,"threads_started":1}
I20260927 00:51:01.634145 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:01.633514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.2:56697)
0927 00:51:01.633886 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:01.633961 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":111,"threads_started":1}
I20260927 00:51:01.823175 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 16907
I20260927 00:51:01.830281 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 17080
I20260927 00:51:01.833979 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:01.833437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.3:41529)
0927 00:51:01.833784 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:01.833852 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":122,"threads_started":1}
I20260927 00:51:01.837126 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 17214
2026-09-27T00:51:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59520 ms)
[----------] 30 tests from SecurityITest (255737 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:51:01 dist-test-slave-86rs krb5kdc[17402](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:51:01 dist-test-slave-86rs krb5kdc[17402](info): set up 2 sockets
Sep 27 00:51:01 dist-test-slave-86rs krb5kdc[17402](info): commencing operation
krb5kdc: starting...
W20260927 00:51:02.888494 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:51:02 dist-test-slave-86rs krb5kdc[17402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:51:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:51:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:51:03.027547 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41395
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:03.160871 17415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:03.161268 17415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:03.161343 17415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:03.165817 17415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:03.165926 17415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:03.165967 17415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:03.166005 17415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:03.166040 17415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:03.172489 17415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:03.173837 17415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:03.175042 17415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:03.184640 17421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:03.184654 17420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:03.184738 17415 server_base.cc:1034] running on GCE node
W20260927 00:51:03.184767 17423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:03.185627 17415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:03.186959 17415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:03.188194 17415 hybrid_clock.cc:648] HybridClock initialized: now 1790470263188182 us; error 32 us; skew 500 ppm
Sep 27 00:51:03 dist-test-slave-86rs krb5kdc[17402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:03.191433 17415 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:51:03.192597 17415 webserver.cc:460] Webserver started at http://127.0.98.62:43539/ using document root <none> and password file <none>
I20260927 00:51:03.193302 17415 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:03.193373 17415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:03.193567 17415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:03.195922 17415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "7b0e4ed378264f3ca92a7a2c21f7b60c"
format_stamp: "Formatted at 2026-09-27 00:51:03 on dist-test-slave-86rs"
server_key: "5d179979e1fac88e284c5b1430f6dbda"
server_key_iv: "810d55a394cd6d3d3945c60622972694"
server_key_version: "encryptionkey@0"
I20260927 00:51:03.196581 17415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "7b0e4ed378264f3ca92a7a2c21f7b60c"
format_stamp: "Formatted at 2026-09-27 00:51:03 on dist-test-slave-86rs"
server_key: "5d179979e1fac88e284c5b1430f6dbda"
server_key_iv: "810d55a394cd6d3d3945c60622972694"
server_key_version: "encryptionkey@0"
I20260927 00:51:03.200877 17415 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:51:03.203572 17430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:03.205145 17415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:51:03.205315 17415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "7b0e4ed378264f3ca92a7a2c21f7b60c"
format_stamp: "Formatted at 2026-09-27 00:51:03 on dist-test-slave-86rs"
server_key: "5d179979e1fac88e284c5b1430f6dbda"
server_key_iv: "810d55a394cd6d3d3945c60622972694"
server_key_version: "encryptionkey@0"
I20260927 00:51:03.205456 17415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:03.221285 17415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:03.224900 17415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:03.225128 17415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:03.233017 17415 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41395
I20260927 00:51:03.233034 17482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41395 every 8 connection(s)
I20260927 00:51:03.234237 17415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:51:03.235754 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 17415
I20260927 00:51:03.235927 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:51:03.236341 392 external_mini_cluster.cc:1428] Setting key 773db353cbd0e2a40266713e1adcf1f0
I20260927 00:51:03.238256 17483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:03.244933 17483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b0e4ed378264f3ca92a7a2c21f7b60c: Bootstrap starting.
Sep 27 00:51:03 dist-test-slave-86rs krb5kdc[17402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:51:03.247651 17483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b0e4ed378264f3ca92a7a2c21f7b60c: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:03.248693 17483 log.cc:824] T 00000000000000000000000000000000 P 7b0e4ed378264f3ca92a7a2c21f7b60c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:03.251377 17483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b0e4ed378264f3ca92a7a2c21f7b60c: No bootstrap required, opened a new log
I20260927 00:51:03.252988 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:03.238059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33950 (local address 127.0.98.62:41395)
0927 00:51:03.238661 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:03.238679 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:03.238725 (+ 46us) server_negotiation.cc:407] Connection header received
0927 00:51:03.239484 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:03.239504 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:03.239905 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:03.240379 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:03.241270 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:03.242335 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:03.243306 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:03.243829 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:03.244018 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:03.246837 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:03.246843 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:03.246858 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:03.246896 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:03.249216 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:03.249836 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:03.249843 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:03.249849 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:03.249994 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:03.250302 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:03.250304 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:03.250305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:03.250754 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:03.250933 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:03.251259 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 00:51:03.251619 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":175,"threads_started":1}
I20260927 00:51:03.255321 17483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b0e4ed378264f3ca92a7a2c21f7b60c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b0e4ed378264f3ca92a7a2c21f7b60c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } }
I20260927 00:51:03.255616 17483 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b0e4ed378264f3ca92a7a2c21f7b60c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:03.255713 17483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b0e4ed378264f3ca92a7a2c21f7b60c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b0e4ed378264f3ca92a7a2c21f7b60c, State: Initialized, Role: FOLLOWER
I20260927 00:51:03.256172 17483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b0e4ed378264f3ca92a7a2c21f7b60c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b0e4ed378264f3ca92a7a2c21f7b60c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } }
I20260927 00:51:03.256424 17483 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b0e4ed378264f3ca92a7a2c21f7b60c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:51:03.256516 17483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b0e4ed378264f3ca92a7a2c21f7b60c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:51:03.256635 17483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b0e4ed378264f3ca92a7a2c21f7b60c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:03.257755 17483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b0e4ed378264f3ca92a7a2c21f7b60c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b0e4ed378264f3ca92a7a2c21f7b60c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } }
I20260927 00:51:03.258147 17483 leader_election.cc:302] T 00000000000000000000000000000000 P 7b0e4ed378264f3ca92a7a2c21f7b60c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b0e4ed378264f3ca92a7a2c21f7b60c; no voters:
I20260927 00:51:03.258540 17483 leader_election.cc:288] T 00000000000000000000000000000000 P 7b0e4ed378264f3ca92a7a2c21f7b60c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:51:03.258709 17488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b0e4ed378264f3ca92a7a2c21f7b60c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:03.258976 17488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b0e4ed378264f3ca92a7a2c21f7b60c [term 1 LEADER]: Becoming Leader. State: Replica: 7b0e4ed378264f3ca92a7a2c21f7b60c, State: Running, Role: LEADER
I20260927 00:51:03.259344 17488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b0e4ed378264f3ca92a7a2c21f7b60c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b0e4ed378264f3ca92a7a2c21f7b60c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } }
I20260927 00:51:03.259920 17483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b0e4ed378264f3ca92a7a2c21f7b60c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:51:03.261174 17490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b0e4ed378264f3ca92a7a2c21f7b60c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b0e4ed378264f3ca92a7a2c21f7b60c. Latest consensus state: current_term: 1 leader_uuid: "7b0e4ed378264f3ca92a7a2c21f7b60c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b0e4ed378264f3ca92a7a2c21f7b60c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } } }
I20260927 00:51:03.261312 17490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b0e4ed378264f3ca92a7a2c21f7b60c [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:03.261754 17489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b0e4ed378264f3ca92a7a2c21f7b60c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b0e4ed378264f3ca92a7a2c21f7b60c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b0e4ed378264f3ca92a7a2c21f7b60c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } } }
I20260927 00:51:03.261952 17489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b0e4ed378264f3ca92a7a2c21f7b60c [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:03.261988 17497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:51:03.265555 17497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:51:03.272390 17497 catalog_manager.cc:1347] Generated new cluster ID: d7f2e49a33e1471bb40beaed61c16c5c
I20260927 00:51:03.272501 17497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:51:03.314553 17497 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:51:03.315649 17497 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:51:03.322412 17497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b0e4ed378264f3ca92a7a2c21f7b60c: Generated new TSK 0
I20260927 00:51:03.323397 17497 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:51:03.360505 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41395
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:03.484771 17509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:03.485087 17509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:03.485175 17509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:03.489542 17509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:03.489614 17509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:03.489720 17509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:51:03.495565 17509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:03.496982 17509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:03.498159 17509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:03.508550 17517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:03.508679 17509 server_base.cc:1034] running on GCE node
W20260927 00:51:03.508508 17514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:03.508579 17515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:03.509279 17509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:03.510056 17509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:03.511250 17509 hybrid_clock.cc:648] HybridClock initialized: now 1790470263511227 us; error 50 us; skew 500 ppm
Sep 27 00:51:03 dist-test-slave-86rs krb5kdc[17402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:03.514353 17509 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:51:03.515398 17509 webserver.cc:460] Webserver started at http://127.0.98.1:46319/ using document root <none> and password file <none>
I20260927 00:51:03.515975 17509 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:03.516037 17509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:03.516208 17509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:03.518527 17509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-0/data/instance:
uuid: "7b8e78508dcc477cbd02d648f6899d5f"
format_stamp: "Formatted at 2026-09-27 00:51:03 on dist-test-slave-86rs"
server_key: "cf5163751d906b0aba036b1767c93e89"
server_key_iv: "48d60b69fafc110ce413feb04cde1575"
server_key_version: "encryptionkey@0"
I20260927 00:51:03.519124 17509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7b8e78508dcc477cbd02d648f6899d5f"
format_stamp: "Formatted at 2026-09-27 00:51:03 on dist-test-slave-86rs"
server_key: "cf5163751d906b0aba036b1767c93e89"
server_key_iv: "48d60b69fafc110ce413feb04cde1575"
server_key_version: "encryptionkey@0"
I20260927 00:51:03.523345 17509 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:51:03.526430 17524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:03.527827 17509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:51:03.528079 17509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "7b8e78508dcc477cbd02d648f6899d5f"
format_stamp: "Formatted at 2026-09-27 00:51:03 on dist-test-slave-86rs"
server_key: "cf5163751d906b0aba036b1767c93e89"
server_key_iv: "48d60b69fafc110ce413feb04cde1575"
server_key_version: "encryptionkey@0"
I20260927 00:51:03.528214 17509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:03.557901 17509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:03.561611 17509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:03.561781 17509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:03.562392 17509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:51:03.563230 17509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:51:03.563279 17509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:03.563320 17509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:51:03.563336 17509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:03.573060 17509 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37819
I20260927 00:51:03.573086 17637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37819 every 8 connection(s)
I20260927 00:51:03.574046 17509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:51:03.579169 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 17509
I20260927 00:51:03.579342 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:51:03.579730 392 external_mini_cluster.cc:1428] Setting key e57b495f37ba41209029413d4de314a3
Sep 27 00:51:03 dist-test-slave-86rs krb5kdc[17402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:51:03.589484 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:03.575959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44411 (local address 127.0.98.62:41395)
0927 00:51:03.576106 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:03.576110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:03.576991 (+ 881us) server_negotiation.cc:407] Connection header received
0927 00:51:03.578232 (+ 1241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:03.578235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:03.578283 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:03.578363 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:03.579725 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:03.580509 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:03.582121 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:03.582396 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:03.582490 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:03.585449 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:03.585452 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:03.585454 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:03.585480 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:03.587106 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:03.587728 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:03.587731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:03.587733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:03.587825 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:03.588198 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:03.588203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:03.588205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:03.588474 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:03.588589 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:03.589142 (+ 553us) server_negotiation.cc:299] Negotiation successful
0927 00:51:03.589291 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:51:03.590425 17640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:03.576325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.1:44411)
0927 00:51:03.576796 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:03.576839 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:03.578023 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:03.578542 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:03.578549 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:03.578962 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:03.579575 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:03.579592 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:03.580657 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:03.580661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:03.581943 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:03.581957 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:03.582577 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:03.582584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:03.582836 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:03.583641 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:51:03.583658 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:03.585236 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:03.587264 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:03.587273 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:03.587292 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:03.587608 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:03.587933 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:03.587936 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:03.587939 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:03.588097 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:03.588588 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:03.588597 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:03.588885 (+ 288us) client_negotiation.cc:769] Sending connection context
0927 00:51:03.589103 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 00:51:03.589470 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":131,"threads_started":1}
I20260927 00:51:03.591969 17638 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41395
I20260927 00:51:03.592312 17638 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:51:03.592969 17638 heartbeater.cc:511] Master 127.0.98.62:41395 requested a full tablet report, sending...
I20260927 00:51:03.594728 17447 ts_manager.cc:194] Registered new tserver with Master: 7b8e78508dcc477cbd02d648f6899d5f (127.0.98.1:37819)
I20260927 00:51:03.596437 17447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44411
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:51:03.611889 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41395
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:03.727939 17643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:03.728282 17643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:03.728350 17643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:03.732697 17643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:03.732769 17643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:03.732836 17643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:51:03.738338 17643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:03.739459 17643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:03.740587 17643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:03.750800 17651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:03.750839 17648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:03.750907 17643 server_base.cc:1034] running on GCE node
W20260927 00:51:03.751121 17649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:03.751569 17643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:03.752410 17643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:03.753585 17643 hybrid_clock.cc:648] HybridClock initialized: now 1790470263753567 us; error 35 us; skew 500 ppm
Sep 27 00:51:03 dist-test-slave-86rs krb5kdc[17402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:03.756620 17643 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:51:03.757592 17643 webserver.cc:460] Webserver started at http://127.0.98.2:40629/ using document root <none> and password file <none>
I20260927 00:51:03.758200 17643 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:03.758296 17643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:03.758543 17643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:03.760977 17643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-1/data/instance:
uuid: "773597abc3974e4b83ad7e7e738aa5ac"
format_stamp: "Formatted at 2026-09-27 00:51:03 on dist-test-slave-86rs"
server_key: "5d9dbb0c333a7f6a47be318965bb0515"
server_key_iv: "93bd142dbddf0115c90eef7481fb8b03"
server_key_version: "encryptionkey@0"
I20260927 00:51:03.761567 17643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-1/wal/instance:
uuid: "773597abc3974e4b83ad7e7e738aa5ac"
format_stamp: "Formatted at 2026-09-27 00:51:03 on dist-test-slave-86rs"
server_key: "5d9dbb0c333a7f6a47be318965bb0515"
server_key_iv: "93bd142dbddf0115c90eef7481fb8b03"
server_key_version: "encryptionkey@0"
I20260927 00:51:03.765655 17643 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:51:03.768311 17658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:03.769631 17643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:51:03.769778 17643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "773597abc3974e4b83ad7e7e738aa5ac"
format_stamp: "Formatted at 2026-09-27 00:51:03 on dist-test-slave-86rs"
server_key: "5d9dbb0c333a7f6a47be318965bb0515"
server_key_iv: "93bd142dbddf0115c90eef7481fb8b03"
server_key_version: "encryptionkey@0"
I20260927 00:51:03.769908 17643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:03.836668 17643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:03.840386 17643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:03.840611 17643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:03.841271 17643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:51:03.842199 17643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:51:03.842283 17643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:03.842375 17643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:51:03.842427 17643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:03.851723 17643 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35987
I20260927 00:51:03.851748 17771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35987 every 8 connection(s)
I20260927 00:51:03.852802 17643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:51:03.860563 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 17643
I20260927 00:51:03.860718 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:51:03.861104 392 external_mini_cluster.cc:1428] Setting key 77b79126191055406d941ba34f912f3f
Sep 27 00:51:03 dist-test-slave-86rs krb5kdc[17402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:51:03.868045 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:03.854815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41839 (local address 127.0.98.62:41395)
0927 00:51:03.854967 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:03.854971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:03.855771 (+ 800us) server_negotiation.cc:407] Connection header received
0927 00:51:03.857166 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:03.857170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:03.857234 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:03.857352 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:03.858594 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:03.859130 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:03.860379 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:03.860611 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:03.860728 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:03.863766 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:03.863769 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:03.863771 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:03.863797 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:03.865616 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:03.866187 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:03.866194 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:03.866199 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:03.866300 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:03.866646 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:03.866653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:03.866658 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:03.866953 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:03.867072 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:03.867623 (+ 551us) server_negotiation.cc:299] Negotiation successful
0927 00:51:03.867782 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:51:03.868924 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:03.855151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.2:41839)
0927 00:51:03.855609 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:03.855653 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:03.856931 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:03.857536 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:03.857544 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:03.857934 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:03.858429 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:03.858443 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:03.859260 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:03.859263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:03.860248 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:03.860257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:03.860775 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:03.860778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:03.861054 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:03.861860 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:51:03.861878 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:03.863575 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:03.865758 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:03.865763 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:03.865780 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:03.866074 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:03.866401 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:03.866403 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:03.866405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:03.866547 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:03.867061 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:03.867066 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:03.867337 (+ 271us) client_negotiation.cc:769] Sending connection context
0927 00:51:03.867533 (+ 196us) client_negotiation.cc:240] Negotiation successful
0927 00:51:03.867860 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":126,"threads_started":1}
I20260927 00:51:03.870272 17772 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41395
I20260927 00:51:03.870563 17772 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:51:03.871223 17772 heartbeater.cc:511] Master 127.0.98.62:41395 requested a full tablet report, sending...
I20260927 00:51:03.872576 17447 ts_manager.cc:194] Registered new tserver with Master: 773597abc3974e4b83ad7e7e738aa5ac (127.0.98.2:35987)
I20260927 00:51:03.873286 17447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41839
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:51:03.891901 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41395
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:04.019698 17777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:04.020030 17777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:04.020079 17777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:04.024503 17777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:04.024590 17777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:04.024657 17777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:51:04.030686 17777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:04.032215 17777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:04.033515 17777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:04.044659 17782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:04.044688 17785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:04.044659 17783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:04.045094 17777 server_base.cc:1034] running on GCE node
I20260927 00:51:04.045598 17777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:04.046325 17777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:04.047534 17777 hybrid_clock.cc:648] HybridClock initialized: now 1790470264047506 us; error 43 us; skew 500 ppm
Sep 27 00:51:04 dist-test-slave-86rs krb5kdc[17402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:04.050700 17777 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:51:04.051839 17777 webserver.cc:460] Webserver started at http://127.0.98.3:41021/ using document root <none> and password file <none>
I20260927 00:51:04.052569 17777 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:04.052673 17777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:04.052929 17777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:04.055171 17777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-2/data/instance:
uuid: "a459e8c32fcd4ac1aabd2f13ec0820be"
format_stamp: "Formatted at 2026-09-27 00:51:04 on dist-test-slave-86rs"
server_key: "18757394c2cf42bac0e795246d5cd412"
server_key_iv: "f0eb39e2ba3abc68a51d1c179f5b9429"
server_key_version: "encryptionkey@0"
I20260927 00:51:04.055817 17777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a459e8c32fcd4ac1aabd2f13ec0820be"
format_stamp: "Formatted at 2026-09-27 00:51:04 on dist-test-slave-86rs"
server_key: "18757394c2cf42bac0e795246d5cd412"
server_key_iv: "f0eb39e2ba3abc68a51d1c179f5b9429"
server_key_version: "encryptionkey@0"
I20260927 00:51:04.060382 17777 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 00:51:04.063633 17792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:04.065191 17777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 00:51:04.065392 17777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "a459e8c32fcd4ac1aabd2f13ec0820be"
format_stamp: "Formatted at 2026-09-27 00:51:04 on dist-test-slave-86rs"
server_key: "18757394c2cf42bac0e795246d5cd412"
server_key_iv: "f0eb39e2ba3abc68a51d1c179f5b9429"
server_key_version: "encryptionkey@0"
I20260927 00:51:04.065551 17777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:04.077003 17777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:04.080881 17777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:04.081130 17777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:04.081818 17777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:51:04.082901 17777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:51:04.082978 17777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:04.083050 17777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:51:04.083098 17777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:04.096095 17777 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37427
I20260927 00:51:04.096112 17905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37427 every 8 connection(s)
I20260927 00:51:04.097363 17777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:51:04.102082 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 17777
I20260927 00:51:04.102216 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:51:04.102622 392 external_mini_cluster.cc:1428] Setting key 325f59bee8e56890eacdbf0e4776fe38
Sep 27 00:51:04 dist-test-slave-86rs krb5kdc[17402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:51:04.113895 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:04.099647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33785 (local address 127.0.98.62:41395)
0927 00:51:04.099824 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:04.099829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:04.100947 (+ 1118us) server_negotiation.cc:407] Connection header received
0927 00:51:04.102416 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:04.102419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:04.102487 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:04.102607 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:04.104148 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.104820 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.106108 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.106339 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.106445 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:04.109345 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:04.109348 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:04.109351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:04.109383 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:04.111088 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:04.111737 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:04.111741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:04.111742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:04.111844 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:04.112353 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:04.112356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:04.112357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:04.112597 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:04.112705 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:04.113375 (+ 670us) server_negotiation.cc:299] Negotiation successful
0927 00:51:04.113645 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:51:04.114547 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:04.100125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.3:33785)
0927 00:51:04.100757 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:04.100809 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:04.102061 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:04.102809 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:04.102816 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:04.103332 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:04.103985 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:04.104007 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.104964 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.104968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:04.105977 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:04.105988 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.106485 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.106488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:04.106643 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:04.107318 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:51:04.107336 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:04.109114 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:04.111246 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:04.111252 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:04.111270 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:04.111597 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:04.111983 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:04.111986 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:04.111992 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:04.112168 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:04.112762 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:04.112772 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:04.113075 (+ 303us) client_negotiation.cc:769] Sending connection context
0927 00:51:04.113259 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 00:51:04.113578 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":169,"threads_started":1}
I20260927 00:51:04.116113 17906 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41395
I20260927 00:51:04.116499 17906 heartbeater.cc:464] Registering TS with master...
I20260927 00:51:04.117179 17906 heartbeater.cc:511] Master 127.0.98.62:41395 requested a full tablet report, sending...
I20260927 00:51:04.118247 17447 ts_manager.cc:194] Registered new tserver with Master: a459e8c32fcd4ac1aabd2f13ec0820be (127.0.98.3:37427)
I20260927 00:51:04.118878 17447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33785
I20260927 00:51:04.129702 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:51:04.140309 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:04.132132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33960 (local address 127.0.98.62:41395)
0927 00:51:04.132339 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:04.132345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:04.132495 (+ 150us) server_negotiation.cc:407] Connection header received
0927 00:51:04.132671 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:04.132674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:04.132728 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:04.132822 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:04.133587 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.134143 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.135056 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.135289 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.135420 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:04.137026 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:04.137030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:04.137032 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:04.137062 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:04.138486 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:04.138947 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:04.138955 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:04.138956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:04.139034 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:04.139314 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:04.139318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:04.139320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:04.139574 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:04.139725 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:04.139979 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 00:51:04.140105 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 00:51:04.143765 17447 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:51:04.147083 17447 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:51:04.163352 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:04.156139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37427 (local address 127.0.0.1:58558)
0927 00:51:04.157716 (+ 1577us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:04.157782 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:04.157980 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:04.158866 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:04.158872 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:04.158889 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:04.158994 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:04.159000 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.160091 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.160095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:04.161648 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:04.161682 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.162726 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.162730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:04.162844 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:04.162889 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 00:51:04.163021 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 00:51:04.163166 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1385,"mutex_wait_us":140,"spinlock_wait_cycles":180992,"thread_start_us":133,"threads_started":1}
I20260927 00:51:04.163362 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:04.156284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58558 (local address 127.0.98.3:37427)
0927 00:51:04.157084 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:04.157096 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:04.157917 (+ 821us) server_negotiation.cc:407] Connection header received
0927 00:51:04.158131 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:04.158137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:04.158365 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:04.158522 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:04.159115 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.159957 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.161789 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.162589 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.162708 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:04.162872 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:04.163073 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 00:51:04.163191 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":115,"threads_started":1}
I20260927 00:51:04.163959 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:04.156466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55092 (local address 127.0.98.1:37819)
0927 00:51:04.156983 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:04.156994 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:04.157896 (+ 902us) server_negotiation.cc:407] Connection header received
0927 00:51:04.158131 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:04.158137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:04.158364 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:04.158522 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:04.159013 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.160848 (+ 1835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.162374 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.163119 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.163217 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:04.163346 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:04.163688 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 00:51:04.163804 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":109,"threads_started":1}
I20260927 00:51:04.164121 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:04.157233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53966 (local address 127.0.98.2:35987)
0927 00:51:04.158180 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:04.158197 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:04.158221 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:51:04.158315 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:04.158319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:04.158533 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:04.158684 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:04.159050 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.160342 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.162186 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.163019 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.163126 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:04.163257 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:04.163868 (+ 611us) server_negotiation.cc:299] Negotiation successful
0927 00:51:04.163971 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":159,"threads_started":1}
I20260927 00:51:04.164319 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:04.156273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37819 (local address 127.0.0.1:55092)
0927 00:51:04.157716 (+ 1443us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:04.157778 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:04.157921 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:04.158533 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:04.158540 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:04.158602 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:04.158803 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:04.158812 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.160995 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.160998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:04.162248 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:04.162255 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.163409 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.163415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:04.163545 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:04.163572 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 00:51:04.163646 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 00:51:04.163711 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1272,"mutex_wait_us":25,"spinlock_wait_cycles":3328,"thread_start_us":121,"threads_started":1}
I20260927 00:51:04.163762 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:04.156924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35987 (local address 127.0.0.1:53966)
0927 00:51:04.157716 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:04.157778 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:04.157921 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:04.158699 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:04.158702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:04.158727 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:04.158883 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:04.158890 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.160507 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.160510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:04.162053 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:04.162064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.163321 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.163325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:04.163436 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:04.163470 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 00:51:04.163547 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 00:51:04.163623 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":55,"spinlock_wait_cycles":1280,"thread_start_us":104,"threads_started":1}
I20260927 00:51:04.167194 17840 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe7aa1508b346b0be4ce038ac896b53 (DEFAULT_TABLE table=test-table [id=ca1b0a803c1a472bb09d39d0962fbc56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:51:04.167361 17572 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe7aa1508b346b0be4ce038ac896b53 (DEFAULT_TABLE table=test-table [id=ca1b0a803c1a472bb09d39d0962fbc56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:51:04.167361 17706 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe7aa1508b346b0be4ce038ac896b53 (DEFAULT_TABLE table=test-table [id=ca1b0a803c1a472bb09d39d0962fbc56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:51:04.168504 17572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fe7aa1508b346b0be4ce038ac896b53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:04.168504 17706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fe7aa1508b346b0be4ce038ac896b53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:04.168507 17840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fe7aa1508b346b0be4ce038ac896b53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:04.175557 17923 tablet_bootstrap.cc:492] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be: Bootstrap starting.
I20260927 00:51:04.176977 17925 tablet_bootstrap.cc:492] T 4fe7aa1508b346b0be4ce038ac896b53 P 773597abc3974e4b83ad7e7e738aa5ac: Bootstrap starting.
I20260927 00:51:04.176977 17924 tablet_bootstrap.cc:492] T 4fe7aa1508b346b0be4ce038ac896b53 P 7b8e78508dcc477cbd02d648f6899d5f: Bootstrap starting.
I20260927 00:51:04.177897 17923 tablet_bootstrap.cc:654] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:04.179082 17923 log.cc:824] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:04.179323 17924 tablet_bootstrap.cc:654] T 4fe7aa1508b346b0be4ce038ac896b53 P 7b8e78508dcc477cbd02d648f6899d5f: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:04.179744 17925 tablet_bootstrap.cc:654] T 4fe7aa1508b346b0be4ce038ac896b53 P 773597abc3974e4b83ad7e7e738aa5ac: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:04.180392 17924 log.cc:824] T 4fe7aa1508b346b0be4ce038ac896b53 P 7b8e78508dcc477cbd02d648f6899d5f: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:04.180647 17925 log.cc:824] T 4fe7aa1508b346b0be4ce038ac896b53 P 773597abc3974e4b83ad7e7e738aa5ac: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:04.181537 17923 tablet_bootstrap.cc:492] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be: No bootstrap required, opened a new log
I20260927 00:51:04.181768 17923 ts_tablet_manager.cc:1397] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:51:04.182545 17924 tablet_bootstrap.cc:492] T 4fe7aa1508b346b0be4ce038ac896b53 P 7b8e78508dcc477cbd02d648f6899d5f: No bootstrap required, opened a new log
I20260927 00:51:04.182542 17925 tablet_bootstrap.cc:492] T 4fe7aa1508b346b0be4ce038ac896b53 P 773597abc3974e4b83ad7e7e738aa5ac: No bootstrap required, opened a new log
I20260927 00:51:04.182735 17924 ts_tablet_manager.cc:1397] T 4fe7aa1508b346b0be4ce038ac896b53 P 7b8e78508dcc477cbd02d648f6899d5f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 00:51:04.182735 17925 ts_tablet_manager.cc:1397] T 4fe7aa1508b346b0be4ce038ac896b53 P 773597abc3974e4b83ad7e7e738aa5ac: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:51:04.185034 17923 raft_consensus.cc:348] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a459e8c32fcd4ac1aabd2f13ec0820be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37427 } } peers { permanent_uuid: "7b8e78508dcc477cbd02d648f6899d5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "773597abc3974e4b83ad7e7e738aa5ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35987 } }
I20260927 00:51:04.185309 17923 raft_consensus.cc:374] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:04.185393 17923 raft_consensus.cc:729] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a459e8c32fcd4ac1aabd2f13ec0820be, State: Initialized, Role: FOLLOWER
I20260927 00:51:04.185886 17923 consensus_queue.cc:260] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a459e8c32fcd4ac1aabd2f13ec0820be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37427 } } peers { permanent_uuid: "7b8e78508dcc477cbd02d648f6899d5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "773597abc3974e4b83ad7e7e738aa5ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35987 } }
I20260927 00:51:04.186794 17925 raft_consensus.cc:348] T 4fe7aa1508b346b0be4ce038ac896b53 P 773597abc3974e4b83ad7e7e738aa5ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a459e8c32fcd4ac1aabd2f13ec0820be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37427 } } peers { permanent_uuid: "7b8e78508dcc477cbd02d648f6899d5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "773597abc3974e4b83ad7e7e738aa5ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35987 } }
I20260927 00:51:04.187000 17923 ts_tablet_manager.cc:1428] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:51:04.187140 17925 raft_consensus.cc:374] T 4fe7aa1508b346b0be4ce038ac896b53 P 773597abc3974e4b83ad7e7e738aa5ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:04.187250 17925 raft_consensus.cc:729] T 4fe7aa1508b346b0be4ce038ac896b53 P 773597abc3974e4b83ad7e7e738aa5ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 773597abc3974e4b83ad7e7e738aa5ac, State: Initialized, Role: FOLLOWER
I20260927 00:51:04.187363 17906 heartbeater.cc:503] Master 127.0.98.62:41395 was elected leader, sending a full tablet report...
I20260927 00:51:04.187219 17924 raft_consensus.cc:348] T 4fe7aa1508b346b0be4ce038ac896b53 P 7b8e78508dcc477cbd02d648f6899d5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a459e8c32fcd4ac1aabd2f13ec0820be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37427 } } peers { permanent_uuid: "7b8e78508dcc477cbd02d648f6899d5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "773597abc3974e4b83ad7e7e738aa5ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35987 } }
I20260927 00:51:04.187454 17924 raft_consensus.cc:374] T 4fe7aa1508b346b0be4ce038ac896b53 P 7b8e78508dcc477cbd02d648f6899d5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:04.187502 17924 raft_consensus.cc:729] T 4fe7aa1508b346b0be4ce038ac896b53 P 7b8e78508dcc477cbd02d648f6899d5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8e78508dcc477cbd02d648f6899d5f, State: Initialized, Role: FOLLOWER
I20260927 00:51:04.187803 17925 consensus_queue.cc:260] T 4fe7aa1508b346b0be4ce038ac896b53 P 773597abc3974e4b83ad7e7e738aa5ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a459e8c32fcd4ac1aabd2f13ec0820be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37427 } } peers { permanent_uuid: "7b8e78508dcc477cbd02d648f6899d5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "773597abc3974e4b83ad7e7e738aa5ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35987 } }
I20260927 00:51:04.187871 17924 consensus_queue.cc:260] T 4fe7aa1508b346b0be4ce038ac896b53 P 7b8e78508dcc477cbd02d648f6899d5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a459e8c32fcd4ac1aabd2f13ec0820be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37427 } } peers { permanent_uuid: "7b8e78508dcc477cbd02d648f6899d5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "773597abc3974e4b83ad7e7e738aa5ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35987 } }
I20260927 00:51:04.188761 17638 heartbeater.cc:503] Master 127.0.98.62:41395 was elected leader, sending a full tablet report...
I20260927 00:51:04.188851 17925 ts_tablet_manager.cc:1428] T 4fe7aa1508b346b0be4ce038ac896b53 P 773597abc3974e4b83ad7e7e738aa5ac: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:51:04.188851 17924 ts_tablet_manager.cc:1428] T 4fe7aa1508b346b0be4ce038ac896b53 P 7b8e78508dcc477cbd02d648f6899d5f: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260927 00:51:04.189821 17772 heartbeater.cc:503] Master 127.0.98.62:41395 was elected leader, sending a full tablet report...
I20260927 00:51:04.282465 17929 raft_consensus.cc:482] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:51:04.282703 17929 raft_consensus.cc:504] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a459e8c32fcd4ac1aabd2f13ec0820be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37427 } } peers { permanent_uuid: "7b8e78508dcc477cbd02d648f6899d5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "773597abc3974e4b83ad7e7e738aa5ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35987 } }
I20260927 00:51:04.283967 17929 leader_election.cc:288] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b8e78508dcc477cbd02d648f6899d5f (127.0.98.1:37819), 773597abc3974e4b83ad7e7e738aa5ac (127.0.98.2:35987)
I20260927 00:51:04.289286 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:04.284499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45213 (local address 127.0.98.1:37819)
0927 00:51:04.284628 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:04.284633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:04.284647 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:51:04.284713 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:04.284715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:04.284756 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:04.284823 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:04.285204 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.286115 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.287999 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.288716 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.288834 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:04.288912 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:04.289106 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 00:51:04.289173 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:51:04.289317 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:04.284300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37819 (local address 127.0.98.3:45213)
0927 00:51:04.284429 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:04.284445 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:04.284543 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:04.284869 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:04.284872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:04.284892 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:04.285031 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:04.285036 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.286256 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.286260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:04.287840 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:04.287846 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.288837 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.288844 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:04.288976 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:04.288998 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:51:04.289079 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 00:51:04.289167 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":118,"spinlock_wait_cycles":768}
I20260927 00:51:04.289433 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:04.284366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39061 (local address 127.0.98.2:35987)
0927 00:51:04.284500 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:04.284507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:04.284889 (+ 382us) server_negotiation.cc:407] Connection header received
0927 00:51:04.284956 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:04.284960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:04.285064 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:04.285142 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:04.285472 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.286237 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.287984 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.288767 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.288877 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:04.288944 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:04.289232 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 00:51:04.289306 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:51:04.289785 17932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:04.284300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35987 (local address 127.0.98.3:39061)
0927 00:51:04.284763 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:04.284774 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:04.284883 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:04.285174 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:04.285177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:04.285196 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:04.285297 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:04.285303 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.286353 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.286358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:04.287834 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:04.287846 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.288995 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.288998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:04.289103 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:04.289121 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 00:51:04.289555 (+ 434us) client_negotiation.cc:240] Negotiation successful
0927 00:51:04.289645 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":210,"thread_start_us":118,"threads_started":1}
I20260927 00:51:04.290104 17592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe7aa1508b346b0be4ce038ac896b53" candidate_uuid: "a459e8c32fcd4ac1aabd2f13ec0820be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b8e78508dcc477cbd02d648f6899d5f" is_pre_election: true
I20260927 00:51:04.290347 17726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe7aa1508b346b0be4ce038ac896b53" candidate_uuid: "a459e8c32fcd4ac1aabd2f13ec0820be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "773597abc3974e4b83ad7e7e738aa5ac" is_pre_election: true
I20260927 00:51:04.290483 17592 raft_consensus.cc:2454] T 4fe7aa1508b346b0be4ce038ac896b53 P 7b8e78508dcc477cbd02d648f6899d5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a459e8c32fcd4ac1aabd2f13ec0820be in term 0.
I20260927 00:51:04.290632 17726 raft_consensus.cc:2454] T 4fe7aa1508b346b0be4ce038ac896b53 P 773597abc3974e4b83ad7e7e738aa5ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a459e8c32fcd4ac1aabd2f13ec0820be in term 0.
I20260927 00:51:04.291013 17794 leader_election.cc:302] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 773597abc3974e4b83ad7e7e738aa5ac, 7b8e78508dcc477cbd02d648f6899d5f, a459e8c32fcd4ac1aabd2f13ec0820be; no voters:
I20260927 00:51:04.291340 17929 raft_consensus.cc:2781] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:51:04.291422 17929 raft_consensus.cc:482] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:51:04.291460 17929 raft_consensus.cc:3037] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:04.292618 17929 raft_consensus.cc:504] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a459e8c32fcd4ac1aabd2f13ec0820be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37427 } } peers { permanent_uuid: "7b8e78508dcc477cbd02d648f6899d5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "773597abc3974e4b83ad7e7e738aa5ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35987 } }
I20260927 00:51:04.293008 17929 leader_election.cc:288] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be [CANDIDATE]: Term 1 election: Requested vote from peers 7b8e78508dcc477cbd02d648f6899d5f (127.0.98.1:37819), 773597abc3974e4b83ad7e7e738aa5ac (127.0.98.2:35987)
I20260927 00:51:04.293509 17726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe7aa1508b346b0be4ce038ac896b53" candidate_uuid: "a459e8c32fcd4ac1aabd2f13ec0820be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "773597abc3974e4b83ad7e7e738aa5ac"
I20260927 00:51:04.293509 17592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe7aa1508b346b0be4ce038ac896b53" candidate_uuid: "a459e8c32fcd4ac1aabd2f13ec0820be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b8e78508dcc477cbd02d648f6899d5f"
I20260927 00:51:04.293655 17592 raft_consensus.cc:3037] T 4fe7aa1508b346b0be4ce038ac896b53 P 7b8e78508dcc477cbd02d648f6899d5f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:04.293654 17726 raft_consensus.cc:3037] T 4fe7aa1508b346b0be4ce038ac896b53 P 773597abc3974e4b83ad7e7e738aa5ac [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:04.295084 17726 raft_consensus.cc:2454] T 4fe7aa1508b346b0be4ce038ac896b53 P 773597abc3974e4b83ad7e7e738aa5ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a459e8c32fcd4ac1aabd2f13ec0820be in term 1.
I20260927 00:51:04.295084 17592 raft_consensus.cc:2454] T 4fe7aa1508b346b0be4ce038ac896b53 P 7b8e78508dcc477cbd02d648f6899d5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a459e8c32fcd4ac1aabd2f13ec0820be in term 1.
I20260927 00:51:04.295519 17793 leader_election.cc:302] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 773597abc3974e4b83ad7e7e738aa5ac, a459e8c32fcd4ac1aabd2f13ec0820be; no voters:
I20260927 00:51:04.295723 17929 raft_consensus.cc:2781] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:04.296030 17929 raft_consensus.cc:686] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be [term 1 LEADER]: Becoming Leader. State: Replica: a459e8c32fcd4ac1aabd2f13ec0820be, State: Running, Role: LEADER
I20260927 00:51:04.296480 17929 consensus_queue.cc:237] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a459e8c32fcd4ac1aabd2f13ec0820be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37427 } } peers { permanent_uuid: "7b8e78508dcc477cbd02d648f6899d5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "773597abc3974e4b83ad7e7e738aa5ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35987 } }
I20260927 00:51:04.299942 17445 catalog_manager.cc:5495] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be reported cstate change: term changed from 0 to 1, leader changed from <none> to a459e8c32fcd4ac1aabd2f13ec0820be (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a459e8c32fcd4ac1aabd2f13ec0820be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a459e8c32fcd4ac1aabd2f13ec0820be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b8e78508dcc477cbd02d648f6899d5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773597abc3974e4b83ad7e7e738aa5ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35987 } health_report { overall_health: UNKNOWN } } }
I20260927 00:51:04.318522 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:04.314803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58562 (local address 127.0.98.3:37427)
0927 00:51:04.314931 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:04.314934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:04.314947 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:51:04.315069 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:04.315071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:04.315129 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:04.315222 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:51:04.315484 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.315979 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.317049 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.317271 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.317385 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:04.317721 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:51:04.318172 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:51:04.318316 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:04.318344 (+ 28us) server_negotiation.cc:299] Negotiation successful
0927 00:51:04.318401 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:51:04.324928 17592 raft_consensus.cc:1261] T 4fe7aa1508b346b0be4ce038ac896b53 P 7b8e78508dcc477cbd02d648f6899d5f [term 1 FOLLOWER]: Refusing update from remote peer a459e8c32fcd4ac1aabd2f13ec0820be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:51:04.324928 17726 raft_consensus.cc:1261] T 4fe7aa1508b346b0be4ce038ac896b53 P 773597abc3974e4b83ad7e7e738aa5ac [term 1 FOLLOWER]: Refusing update from remote peer a459e8c32fcd4ac1aabd2f13ec0820be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 00:51:04.325578 17639 tablet.cc:2369] T 4fe7aa1508b346b0be4ce038ac896b53 P 7b8e78508dcc477cbd02d648f6899d5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:51:04.325836 17933 consensus_queue.cc:1035] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be [LEADER]: Connected to new peer: Peer: permanent_uuid: "773597abc3974e4b83ad7e7e738aa5ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:51:04.326195 17929 consensus_queue.cc:1035] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b8e78508dcc477cbd02d648f6899d5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:51:04.333977 17937 mvcc.cc:205] Tried to move back new op lower bound from 7333766202666774528 to 7333766202561413120. Current Snapshot: MvccSnapshot[applied={T|T < 7333766202666774528}]
I20260927 00:51:04.335209 17938 mvcc.cc:205] Tried to move back new op lower bound from 7333766202666774528 to 7333766202561413120. Current Snapshot: MvccSnapshot[applied={T|T < 7333766202666774528}]
I20260927 00:51:04.342566 17445 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:51:04.342779 17445 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:51:04.345609 17445 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b0e4ed378264f3ca92a7a2c21f7b60c: Sending DeleteTablet for 3 replicas of tablet 4fe7aa1508b346b0be4ce038ac896b53
I20260927 00:51:04.346517 17572 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fe7aa1508b346b0be4ce038ac896b53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:51:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55092
I20260927 00:51:04.346611 17840 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fe7aa1508b346b0be4ce038ac896b53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:51:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58558
I20260927 00:51:04.346640 17706 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fe7aa1508b346b0be4ce038ac896b53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:51:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53966
I20260927 00:51:04.347289 17946 tablet_replica.cc:330] T 4fe7aa1508b346b0be4ce038ac896b53 P 7b8e78508dcc477cbd02d648f6899d5f: stopping tablet replica
I20260927 00:51:04.347383 17948 tablet_replica.cc:330] T 4fe7aa1508b346b0be4ce038ac896b53 P 773597abc3974e4b83ad7e7e738aa5ac: stopping tablet replica
I20260927 00:51:04.347558 17947 tablet_replica.cc:330] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be: stopping tablet replica
I20260927 00:51:04.347904 17947 raft_consensus.cc:2229] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:51:04.347993 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 17509
I20260927 00:51:04.348022 17948 raft_consensus.cc:2229] T 4fe7aa1508b346b0be4ce038ac896b53 P 773597abc3974e4b83ad7e7e738aa5ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:51:04.348937 17947 raft_consensus.cc:2258] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:51:04.349713 17948 raft_consensus.cc:2258] T 4fe7aa1508b346b0be4ce038ac896b53 P 773597abc3974e4b83ad7e7e738aa5ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:51:04.351182 17947 ts_tablet_manager.cc:1905] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:51:04.353597 17947 ts_tablet_manager.cc:1918] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:51:04.353720 17947 log.cc:1196] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790470006085796-392-0/minicluster-data/ts-2/wal/wals/4fe7aa1508b346b0be4ce038ac896b53
I20260927 00:51:04.354071 17947 ts_tablet_manager.cc:1939] T 4fe7aa1508b346b0be4ce038ac896b53 P a459e8c32fcd4ac1aabd2f13ec0820be: Deleting consensus metadata
I20260927 00:51:04.355361 17948 ts_tablet_manager.cc:1905] T 4fe7aa1508b346b0be4ce038ac896b53 P 773597abc3974e4b83ad7e7e738aa5ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:51:04.355484 17433 catalog_manager.cc:4841] TS a459e8c32fcd4ac1aabd2f13ec0820be (127.0.98.3:37427): tablet 4fe7aa1508b346b0be4ce038ac896b53 (table test-table [id=ca1b0a803c1a472bb09d39d0962fbc56]) successfully deleted
W20260927 00:51:04.356796 17432 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37819 (error 108)
I20260927 00:51:04.357924 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:04.357602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37819 (local address 127.0.0.1:55108)
0927 00:51:04.357746 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:04.357823 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 00:51:04.357949 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 17643
W20260927 00:51:04.358820 17432 catalog_manager.cc:4584] TS 7b8e78508dcc477cbd02d648f6899d5f (127.0.98.1:37819): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4fe7aa1508b346b0be4ce038ac896b53: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37819: connect: Connection refused (error 111)
I20260927 00:51:04.364506 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 17777
I20260927 00:51:04.364781 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:04.364411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35987 (local address 127.0.0.1:53968)
0927 00:51:04.364582 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:04.364664 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 00:51:04.370512 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 17415
2026-09-27T00:51:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2532 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T00:51:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:51:04Z Disabled control of system clock
I20260927 00:51:04.403340 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38283
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:51:04.521375 17953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:04.521682 17953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:04.521718 17953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:04.525941 17953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:04.526021 17953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:04.526036 17953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:04.526054 17953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:04.526069 17953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:04.531207 17953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38283
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:04.532439 17953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:04.533874 17953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:04.542685 17958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:04.542724 17961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:04.542680 17959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:04.542961 17953 server_base.cc:1034] running on GCE node
I20260927 00:51:04.543715 17953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:04.544927 17953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:04.546113 17953 hybrid_clock.cc:648] HybridClock initialized: now 1790470264546083 us; error 42 us; skew 500 ppm
I20260927 00:51:04.548118 17953 webserver.cc:460] Webserver started at http://127.0.98.62:43223/ using document root <none> and password file <none>
I20260927 00:51:04.548777 17953 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:04.548849 17953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:04.549072 17953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:04.551282 17953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "03fb757c36cc459887d5a72927899c3e"
format_stamp: "Formatted at 2026-09-27 00:51:04 on dist-test-slave-86rs"
server_key: "a74d3e5cc2fc8759e623a41adbe6a4ee"
server_key_iv: "46f5688193c37b1d3bbf4712a3ab2419"
server_key_version: "encryptionkey@0"
I20260927 00:51:04.551831 17953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "03fb757c36cc459887d5a72927899c3e"
format_stamp: "Formatted at 2026-09-27 00:51:04 on dist-test-slave-86rs"
server_key: "a74d3e5cc2fc8759e623a41adbe6a4ee"
server_key_iv: "46f5688193c37b1d3bbf4712a3ab2419"
server_key_version: "encryptionkey@0"
I20260927 00:51:04.555737 17953 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:51:04.558367 17967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:04.559783 17953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:51:04.559906 17953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "03fb757c36cc459887d5a72927899c3e"
format_stamp: "Formatted at 2026-09-27 00:51:04 on dist-test-slave-86rs"
server_key: "a74d3e5cc2fc8759e623a41adbe6a4ee"
server_key_iv: "46f5688193c37b1d3bbf4712a3ab2419"
server_key_version: "encryptionkey@0"
I20260927 00:51:04.560010 17953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:04.581672 17953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:04.582434 17953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:04.582602 17953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:04.590034 17953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38283
I20260927 00:51:04.590062 18019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38283 every 8 connection(s)
I20260927 00:51:04.591073 17953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:51:04.594542 18020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:04.600175 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 17953
I20260927 00:51:04.600394 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:51:04.600684 392 external_mini_cluster.cc:1428] Setting key 8d671476e8d6ad73cc098e30f1cc8ec4
I20260927 00:51:04.601243 18020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03fb757c36cc459887d5a72927899c3e: Bootstrap starting.
I20260927 00:51:04.603765 18020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03fb757c36cc459887d5a72927899c3e: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:04.604691 18020 log.cc:824] T 00000000000000000000000000000000 P 03fb757c36cc459887d5a72927899c3e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:04.606988 18020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03fb757c36cc459887d5a72927899c3e: No bootstrap required, opened a new log
I20260927 00:51:04.610714 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:04.602326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53408 (local address 127.0.98.62:38283)
0927 00:51:04.603017 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:04.603032 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:04.603071 (+ 39us) server_negotiation.cc:407] Connection header received
0927 00:51:04.603765 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:04.603783 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:04.604339 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:04.604651 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:04.605075 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.606043 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.607238 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.607728 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.607849 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:04.608822 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:04.608828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:04.608842 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:04.609254 (+ 412us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:04.609314 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:04.609334 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:04.609440 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:04.609581 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 00:51:04.609920 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":161,"threads_started":1}
I20260927 00:51:04.611327 18020 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03fb757c36cc459887d5a72927899c3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03fb757c36cc459887d5a72927899c3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38283 } }
I20260927 00:51:04.611599 18020 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03fb757c36cc459887d5a72927899c3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:04.611686 18020 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03fb757c36cc459887d5a72927899c3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03fb757c36cc459887d5a72927899c3e, State: Initialized, Role: FOLLOWER
I20260927 00:51:04.612217 18020 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03fb757c36cc459887d5a72927899c3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03fb757c36cc459887d5a72927899c3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38283 } }
I20260927 00:51:04.612409 18020 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03fb757c36cc459887d5a72927899c3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:51:04.612486 18020 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03fb757c36cc459887d5a72927899c3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:51:04.612581 18020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03fb757c36cc459887d5a72927899c3e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:04.613727 18020 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03fb757c36cc459887d5a72927899c3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03fb757c36cc459887d5a72927899c3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38283 } }
I20260927 00:51:04.614038 18020 leader_election.cc:302] T 00000000000000000000000000000000 P 03fb757c36cc459887d5a72927899c3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03fb757c36cc459887d5a72927899c3e; no voters:
I20260927 00:51:04.614300 18020 leader_election.cc:288] T 00000000000000000000000000000000 P 03fb757c36cc459887d5a72927899c3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:51:04.614446 18025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03fb757c36cc459887d5a72927899c3e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:04.614709 18025 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03fb757c36cc459887d5a72927899c3e [term 1 LEADER]: Becoming Leader. State: Replica: 03fb757c36cc459887d5a72927899c3e, State: Running, Role: LEADER
I20260927 00:51:04.615099 18025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03fb757c36cc459887d5a72927899c3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03fb757c36cc459887d5a72927899c3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38283 } }
I20260927 00:51:04.615639 18020 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03fb757c36cc459887d5a72927899c3e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:51:04.616931 18026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03fb757c36cc459887d5a72927899c3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03fb757c36cc459887d5a72927899c3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03fb757c36cc459887d5a72927899c3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38283 } } }
I20260927 00:51:04.617075 18026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03fb757c36cc459887d5a72927899c3e [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:04.616909 18027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03fb757c36cc459887d5a72927899c3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03fb757c36cc459887d5a72927899c3e. Latest consensus state: current_term: 1 leader_uuid: "03fb757c36cc459887d5a72927899c3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03fb757c36cc459887d5a72927899c3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38283 } } }
I20260927 00:51:04.617364 18027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03fb757c36cc459887d5a72927899c3e [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:04.617772 18034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:51:04.621160 18034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:51:04.627231 18034 catalog_manager.cc:1347] Generated new cluster ID: 8aa4d60e641a4d12b3b4f764166645fd
I20260927 00:51:04.627327 18034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:51:04.647780 18034 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:51:04.649221 18034 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:51:04.663614 18034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03fb757c36cc459887d5a72927899c3e: Generated new TSK 0
I20260927 00:51:04.664619 18034 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:51:04.670552 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38283
--builtin_ntp_servers=127.0.98.20:45115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:51:04.788352 18044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:04.788636 18044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:04.788676 18044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:04.792788 18044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:04.792857 18044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:04.792917 18044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:51:04.797966 18044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:04.799124 18044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:04.801102 18044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:04.811282 18050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:04.811295 18049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:04.811286 18052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:04.811642 18044 server_base.cc:1034] running on GCE node
I20260927 00:51:04.812093 18044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:04.812865 18044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:04.814055 18044 hybrid_clock.cc:648] HybridClock initialized: now 1790470264814036 us; error 30 us; skew 500 ppm
I20260927 00:51:04.816304 18044 webserver.cc:460] Webserver started at http://127.0.98.1:44109/ using document root <none> and password file <none>
I20260927 00:51:04.816987 18044 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:04.817102 18044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:04.817343 18044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:04.819528 18044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-0/data/instance:
uuid: "1dd66d4d49564484bc18587e81d4d9ad"
format_stamp: "Formatted at 2026-09-27 00:51:04 on dist-test-slave-86rs"
server_key: "d18c3a0cd4ce46237c7a17bd5d75ea5a"
server_key_iv: "d007711653dd683d626c3e459ef229e9"
server_key_version: "encryptionkey@0"
I20260927 00:51:04.820073 18044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1dd66d4d49564484bc18587e81d4d9ad"
format_stamp: "Formatted at 2026-09-27 00:51:04 on dist-test-slave-86rs"
server_key: "d18c3a0cd4ce46237c7a17bd5d75ea5a"
server_key_iv: "d007711653dd683d626c3e459ef229e9"
server_key_version: "encryptionkey@0"
I20260927 00:51:04.823830 18044 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:51:04.826397 18058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:04.827749 18044 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:51:04.827911 18044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "1dd66d4d49564484bc18587e81d4d9ad"
format_stamp: "Formatted at 2026-09-27 00:51:04 on dist-test-slave-86rs"
server_key: "d18c3a0cd4ce46237c7a17bd5d75ea5a"
server_key_iv: "d007711653dd683d626c3e459ef229e9"
server_key_version: "encryptionkey@0"
I20260927 00:51:04.828043 18044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:04.846817 18044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:04.847667 18044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:04.847890 18044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:04.848579 18044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:51:04.849650 18044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:51:04.849725 18044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:04.849797 18044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:51:04.849848 18044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:04.858987 18044 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41121
I20260927 00:51:04.859012 18171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41121 every 8 connection(s)
I20260927 00:51:04.859948 18044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:51:04.867638 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 18044
I20260927 00:51:04.867789 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:51:04.868144 392 external_mini_cluster.cc:1428] Setting key fba61026fee46c0956503d97775fc070
I20260927 00:51:04.869750 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:04.861900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41401 (local address 127.0.98.62:38283)
0927 00:51:04.862045 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:04.862049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:04.863145 (+ 1096us) server_negotiation.cc:407] Connection header received
0927 00:51:04.864684 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:04.864687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:04.864747 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:04.864857 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:04.865797 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.866340 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.867524 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.867786 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.867906 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:04.869246 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:04.869251 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:04.869260 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:04.869273 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:04.869291 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:04.869300 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:04.869381 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:04.869545 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 00:51:04.869608 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:51:04.870218 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38283
--builtin_ntp_servers=127.0.98.20:45115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:51:04.870713 18174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:04.862313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38283 (local address 127.0.98.1:41401)
0927 00:51:04.862984 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:04.863026 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:04.864424 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:04.865047 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:04.865054 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:04.865143 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:04.865645 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:04.865659 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.866499 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.866502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:04.867392 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:04.867403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:04.867996 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:04.868002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:04.868254 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:04.868975 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:51:04.868977 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:04.869020 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:51:04.869088 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:04.869383 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:04.869395 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:04.869402 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:51:04.869545 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 00:51:04.869849 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":112,"threads_started":1}
I20260927 00:51:04.872460 18172 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38283
I20260927 00:51:04.872754 18172 heartbeater.cc:464] Registering TS with master...
I20260927 00:51:04.873445 18172 heartbeater.cc:511] Master 127.0.98.62:38283 requested a full tablet report, sending...
I20260927 00:51:04.875191 17984 ts_manager.cc:194] Registered new tserver with Master: 1dd66d4d49564484bc18587e81d4d9ad (127.0.98.1:41121)
I20260927 00:51:04.876609 17984 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:41401
W20260927 00:51:05.004809 18175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:05.005260 18175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:05.005326 18175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:05.010228 18175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:05.010306 18175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:05.010368 18175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:51:05.015661 18175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:05.017023 18175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:05.018549 18175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:05.028764 18181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:05.028750 18180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:05.028769 18183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:05.029145 18175 server_base.cc:1034] running on GCE node
I20260927 00:51:05.029563 18175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:05.030272 18175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:05.031438 18175 hybrid_clock.cc:648] HybridClock initialized: now 1790470265031420 us; error 31 us; skew 500 ppm
I20260927 00:51:05.033522 18175 webserver.cc:460] Webserver started at http://127.0.98.2:35439/ using document root <none> and password file <none>
I20260927 00:51:05.034157 18175 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:05.034222 18175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:05.034451 18175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:05.036775 18175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-1/data/instance:
uuid: "1c60ef0c03bc4486827edaf45948564f"
format_stamp: "Formatted at 2026-09-27 00:51:05 on dist-test-slave-86rs"
server_key: "693c3d9f3785afd0d5f38ff12c2b16d8"
server_key_iv: "27dc284842caec948e363f0e02528970"
server_key_version: "encryptionkey@0"
I20260927 00:51:05.037402 18175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1c60ef0c03bc4486827edaf45948564f"
format_stamp: "Formatted at 2026-09-27 00:51:05 on dist-test-slave-86rs"
server_key: "693c3d9f3785afd0d5f38ff12c2b16d8"
server_key_iv: "27dc284842caec948e363f0e02528970"
server_key_version: "encryptionkey@0"
I20260927 00:51:05.041337 18175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:51:05.044039 18189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:05.045408 18175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:51:05.045562 18175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "1c60ef0c03bc4486827edaf45948564f"
format_stamp: "Formatted at 2026-09-27 00:51:05 on dist-test-slave-86rs"
server_key: "693c3d9f3785afd0d5f38ff12c2b16d8"
server_key_iv: "27dc284842caec948e363f0e02528970"
server_key_version: "encryptionkey@0"
I20260927 00:51:05.045691 18175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:05.058146 18175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:05.058866 18175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:05.059072 18175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:05.059679 18175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:51:05.060753 18175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:51:05.060823 18175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:05.060896 18175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:51:05.060949 18175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:05.072405 18175 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43023
I20260927 00:51:05.072463 18302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43023 every 8 connection(s)
I20260927 00:51:05.073686 18175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:51:05.077913 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 18175
I20260927 00:51:05.078019 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:51:05.078329 392 external_mini_cluster.cc:1428] Setting key 431617b51daf85faffd9a5db06013cf2
I20260927 00:51:05.080376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38283
--builtin_ntp_servers=127.0.98.20:45115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:51:05.084303 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:05.075740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40157 (local address 127.0.98.62:38283)
0927 00:51:05.075894 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:05.075904 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:05.076987 (+ 1083us) server_negotiation.cc:407] Connection header received
0927 00:51:05.078441 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:05.078444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:05.078508 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:05.078592 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:05.079672 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.080282 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.082106 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.082399 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.082520 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:05.083737 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:05.083741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:05.083744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:05.083758 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:05.083794 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:05.083802 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:05.083881 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:05.084048 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 00:51:05.084116 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:51:05.085364 18305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:05.076076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38283 (local address 127.0.98.2:40157)
0927 00:51:05.076809 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:05.076851 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:05.078211 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:05.078769 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:05.078776 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:05.078894 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:05.079517 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:05.079539 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.080418 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.080423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:05.081954 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:05.081965 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.082568 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.082572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:05.082767 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:05.083448 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:51:05.083450 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:05.083496 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:51:05.083568 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:05.083895 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:05.083905 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:05.083911 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:51:05.084049 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 00:51:05.084405 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":220,"threads_started":1}
I20260927 00:51:05.086616 18303 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38283
I20260927 00:51:05.086901 18303 heartbeater.cc:464] Registering TS with master...
I20260927 00:51:05.087455 18303 heartbeater.cc:511] Master 127.0.98.62:38283 requested a full tablet report, sending...
I20260927 00:51:05.088656 17984 ts_manager.cc:194] Registered new tserver with Master: 1c60ef0c03bc4486827edaf45948564f (127.0.98.2:43023)
I20260927 00:51:05.089584 17984 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:40157
W20260927 00:51:05.197628 18306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:05.197990 18306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:05.198035 18306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:05.202349 18306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:05.202426 18306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:05.202487 18306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:51:05.207567 18306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:05.208887 18306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:05.210274 18306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:05.220239 18314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:05.220358 18312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:05.220219 18311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:05.220441 18306 server_base.cc:1034] running on GCE node
I20260927 00:51:05.221016 18306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:05.221706 18306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:05.222898 18306 hybrid_clock.cc:648] HybridClock initialized: now 1790470265222861 us; error 55 us; skew 500 ppm
I20260927 00:51:05.225014 18306 webserver.cc:460] Webserver started at http://127.0.98.3:42855/ using document root <none> and password file <none>
I20260927 00:51:05.225703 18306 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:05.225811 18306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:05.226059 18306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:05.228508 18306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-2/data/instance:
uuid: "5b39d421768a4a5fb248244a6aebee00"
format_stamp: "Formatted at 2026-09-27 00:51:05 on dist-test-slave-86rs"
server_key: "e964892389596bd7adf828a488fbfbb1"
server_key_iv: "720612fdae10677a2488468f769e7cb2"
server_key_version: "encryptionkey@0"
I20260927 00:51:05.229154 18306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5b39d421768a4a5fb248244a6aebee00"
format_stamp: "Formatted at 2026-09-27 00:51:05 on dist-test-slave-86rs"
server_key: "e964892389596bd7adf828a488fbfbb1"
server_key_iv: "720612fdae10677a2488468f769e7cb2"
server_key_version: "encryptionkey@0"
I20260927 00:51:05.233594 18306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:51:05.236491 18320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:05.237906 18306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:51:05.238067 18306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "5b39d421768a4a5fb248244a6aebee00"
format_stamp: "Formatted at 2026-09-27 00:51:05 on dist-test-slave-86rs"
server_key: "e964892389596bd7adf828a488fbfbb1"
server_key_iv: "720612fdae10677a2488468f769e7cb2"
server_key_version: "encryptionkey@0"
I20260927 00:51:05.238210 18306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:05.285837 18306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:05.286666 18306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:05.286895 18306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:05.287557 18306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:51:05.288592 18306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:51:05.288678 18306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:05.288757 18306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:51:05.288792 18306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:05.299100 18306 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37981
I20260927 00:51:05.299175 18433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37981 every 8 connection(s)
I20260927 00:51:05.300146 18306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:51:05.308198 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 18306
I20260927 00:51:05.308374 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:51:05.308683 392 external_mini_cluster.cc:1428] Setting key c34ea309a37341fd87d2028ea2d1d19b
I20260927 00:51:05.309645 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:05.302155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37843 (local address 127.0.98.62:38283)
0927 00:51:05.302307 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:05.302311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:05.303338 (+ 1027us) server_negotiation.cc:407] Connection header received
0927 00:51:05.304835 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:05.304838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:05.304892 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:05.304973 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:05.305879 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.306399 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.307525 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.307749 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.307832 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:05.309119 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:05.309122 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:05.309124 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:05.309137 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:05.309157 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:05.309165 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:05.309256 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:05.309437 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 00:51:05.309502 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:51:05.310597 18436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:05.302567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38283 (local address 127.0.98.3:37843)
0927 00:51:05.303202 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:05.303245 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:05.304621 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:05.305148 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:05.305154 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:05.305241 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:05.305752 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:05.305765 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.306528 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.306530 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:05.307404 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:05.307412 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.307889 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.307891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:05.308033 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:05.308810 (+ 777us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:51:05.308813 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:05.308872 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:51:05.308944 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:05.309261 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:05.309272 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:05.309278 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:51:05.309419 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 00:51:05.309722 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":108,"threads_started":1}
I20260927 00:51:05.311658 18434 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38283
I20260927 00:51:05.311934 18434 heartbeater.cc:464] Registering TS with master...
I20260927 00:51:05.312552 18434 heartbeater.cc:511] Master 127.0.98.62:38283 requested a full tablet report, sending...
I20260927 00:51:05.313534 17984 ts_manager.cc:194] Registered new tserver with Master: 5b39d421768a4a5fb248244a6aebee00 (127.0.98.3:37981)
I20260927 00:51:05.314108 17984 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:37843
I20260927 00:51:05.322806 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:51:05.328830 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:05.324961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53420 (local address 127.0.98.62:38283)
0927 00:51:05.325115 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:05.325120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:05.325294 (+ 174us) server_negotiation.cc:407] Connection header received
0927 00:51:05.325398 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:05.325400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:05.325445 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:05.325515 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:05.325782 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.326313 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.327135 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.327352 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.327441 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:05.328335 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:05.328338 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:05.328340 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:05.328352 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:05.328369 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:05.328377 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:05.328436 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:05.328547 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 00:51:05.328599 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:51:05.332305 17984 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:51:05.335539 17984 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:51:05.351176 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:05.345421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37981 (local address 127.0.0.1:49492)
0927 00:51:05.345812 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:05.345826 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:05.345908 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:05.346440 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:05.346446 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:05.346469 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:05.346636 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:05.346644 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.348028 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.348031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:05.349615 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:05.349626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.350566 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.350569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:05.350701 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:05.350743 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 00:51:05.350859 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 00:51:05.350993 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":108,"thread_start_us":110,"threads_started":1}
I20260927 00:51:05.351179 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:05.345569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49492 (local address 127.0.98.3:37981)
0927 00:51:05.345932 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:05.345944 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:05.345968 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:51:05.346049 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:05.346056 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:05.346270 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:05.346435 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:05.346759 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.347925 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.349749 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.350452 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.350558 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:05.350765 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:05.350890 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 00:51:05.350992 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":146,"threads_started":1}
I20260927 00:51:05.351229 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:05.344658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43023 (local address 127.0.0.1:38326)
0927 00:51:05.345228 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:05.345287 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:05.345533 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:05.347276 (+ 1743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:05.347279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:05.347297 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:05.347387 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:05.347392 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.348595 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.348600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:05.349666 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:05.349675 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.350795 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.350799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:05.350898 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:05.350921 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:51:05.350976 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 00:51:05.351049 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":99,"threads_started":1}
I20260927 00:51:05.351728 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:05.345645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49582 (local address 127.0.98.1:41121)
0927 00:51:05.346146 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:05.346152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:05.346174 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:51:05.346254 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:05.346266 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:05.346474 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:05.346623 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:05.346989 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.347854 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.349782 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.350657 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.350772 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:05.350960 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:05.351467 (+ 507us) server_negotiation.cc:299] Negotiation successful
0927 00:51:05.351566 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":114,"threads_started":1}
I20260927 00:51:05.351380 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:05.345079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41121 (local address 127.0.0.1:49582)
0927 00:51:05.345358 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:05.345880 (+ 522us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:05.345988 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:05.346780 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:05.346789 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:05.346804 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:05.346873 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:05.346877 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.347970 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.347975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:05.349622 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:05.349627 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.351016 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.351019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:05.351099 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:05.351119 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 00:51:05.351181 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 00:51:05.351241 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":145,"thread_start_us":116,"threads_started":1}
I20260927 00:51:05.351801 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:05.344841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38326 (local address 127.0.98.2:43023)
0927 00:51:05.346844 (+ 2003us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:05.346854 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:05.346876 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:51:05.346944 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:05.346948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:05.347119 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:05.347266 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:05.347502 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.348489 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.349793 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.350657 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.351333 (+ 676us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:05.351517 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:05.351561 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 00:51:05.351643 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1891,"thread_start_us":128,"threads_started":1}
I20260927 00:51:05.354589 18368 tablet_service.cc:1467] Processing CreateTablet for tablet 9954ff2f7f154c0da5c26f4615695d91 (DEFAULT_TABLE table=test-table [id=c082ff91e94141ab9b5ac93dbf6a1aa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:51:05.354717 18106 tablet_service.cc:1467] Processing CreateTablet for tablet 9954ff2f7f154c0da5c26f4615695d91 (DEFAULT_TABLE table=test-table [id=c082ff91e94141ab9b5ac93dbf6a1aa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:51:05.354717 18237 tablet_service.cc:1467] Processing CreateTablet for tablet 9954ff2f7f154c0da5c26f4615695d91 (DEFAULT_TABLE table=test-table [id=c082ff91e94141ab9b5ac93dbf6a1aa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:51:05.355669 18368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9954ff2f7f154c0da5c26f4615695d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:05.355751 18237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9954ff2f7f154c0da5c26f4615695d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:05.355751 18106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9954ff2f7f154c0da5c26f4615695d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:05.362479 18451 tablet_bootstrap.cc:492] T 9954ff2f7f154c0da5c26f4615695d91 P 5b39d421768a4a5fb248244a6aebee00: Bootstrap starting.
I20260927 00:51:05.363722 18453 tablet_bootstrap.cc:492] T 9954ff2f7f154c0da5c26f4615695d91 P 1dd66d4d49564484bc18587e81d4d9ad: Bootstrap starting.
I20260927 00:51:05.363819 18452 tablet_bootstrap.cc:492] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f: Bootstrap starting.
I20260927 00:51:05.365123 18451 tablet_bootstrap.cc:654] T 9954ff2f7f154c0da5c26f4615695d91 P 5b39d421768a4a5fb248244a6aebee00: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:05.365825 18452 tablet_bootstrap.cc:654] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:05.365947 18453 tablet_bootstrap.cc:654] T 9954ff2f7f154c0da5c26f4615695d91 P 1dd66d4d49564484bc18587e81d4d9ad: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:05.366016 18451 log.cc:824] T 9954ff2f7f154c0da5c26f4615695d91 P 5b39d421768a4a5fb248244a6aebee00: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:05.366636 18452 log.cc:824] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:05.366689 18453 log.cc:824] T 9954ff2f7f154c0da5c26f4615695d91 P 1dd66d4d49564484bc18587e81d4d9ad: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:05.368355 18451 tablet_bootstrap.cc:492] T 9954ff2f7f154c0da5c26f4615695d91 P 5b39d421768a4a5fb248244a6aebee00: No bootstrap required, opened a new log
I20260927 00:51:05.368568 18451 ts_tablet_manager.cc:1397] T 9954ff2f7f154c0da5c26f4615695d91 P 5b39d421768a4a5fb248244a6aebee00: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:51:05.368619 18453 tablet_bootstrap.cc:492] T 9954ff2f7f154c0da5c26f4615695d91 P 1dd66d4d49564484bc18587e81d4d9ad: No bootstrap required, opened a new log
I20260927 00:51:05.368698 18452 tablet_bootstrap.cc:492] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f: No bootstrap required, opened a new log
I20260927 00:51:05.368773 18453 ts_tablet_manager.cc:1397] T 9954ff2f7f154c0da5c26f4615695d91 P 1dd66d4d49564484bc18587e81d4d9ad: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:51:05.368880 18452 ts_tablet_manager.cc:1397] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:51:05.372383 18452 raft_consensus.cc:348] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c60ef0c03bc4486827edaf45948564f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43023 } } peers { permanent_uuid: "5b39d421768a4a5fb248244a6aebee00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37981 } } peers { permanent_uuid: "1dd66d4d49564484bc18587e81d4d9ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41121 } }
I20260927 00:51:05.372682 18452 raft_consensus.cc:374] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:05.372759 18452 raft_consensus.cc:729] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c60ef0c03bc4486827edaf45948564f, State: Initialized, Role: FOLLOWER
I20260927 00:51:05.373063 18453 raft_consensus.cc:348] T 9954ff2f7f154c0da5c26f4615695d91 P 1dd66d4d49564484bc18587e81d4d9ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c60ef0c03bc4486827edaf45948564f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43023 } } peers { permanent_uuid: "5b39d421768a4a5fb248244a6aebee00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37981 } } peers { permanent_uuid: "1dd66d4d49564484bc18587e81d4d9ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41121 } }
I20260927 00:51:05.373063 18451 raft_consensus.cc:348] T 9954ff2f7f154c0da5c26f4615695d91 P 5b39d421768a4a5fb248244a6aebee00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c60ef0c03bc4486827edaf45948564f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43023 } } peers { permanent_uuid: "5b39d421768a4a5fb248244a6aebee00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37981 } } peers { permanent_uuid: "1dd66d4d49564484bc18587e81d4d9ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41121 } }
I20260927 00:51:05.373322 18451 raft_consensus.cc:374] T 9954ff2f7f154c0da5c26f4615695d91 P 5b39d421768a4a5fb248244a6aebee00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:05.373366 18453 raft_consensus.cc:374] T 9954ff2f7f154c0da5c26f4615695d91 P 1dd66d4d49564484bc18587e81d4d9ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:05.373273 18452 consensus_queue.cc:260] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c60ef0c03bc4486827edaf45948564f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43023 } } peers { permanent_uuid: "5b39d421768a4a5fb248244a6aebee00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37981 } } peers { permanent_uuid: "1dd66d4d49564484bc18587e81d4d9ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41121 } }
I20260927 00:51:05.373404 18451 raft_consensus.cc:729] T 9954ff2f7f154c0da5c26f4615695d91 P 5b39d421768a4a5fb248244a6aebee00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b39d421768a4a5fb248244a6aebee00, State: Initialized, Role: FOLLOWER
I20260927 00:51:05.373477 18453 raft_consensus.cc:729] T 9954ff2f7f154c0da5c26f4615695d91 P 1dd66d4d49564484bc18587e81d4d9ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dd66d4d49564484bc18587e81d4d9ad, State: Initialized, Role: FOLLOWER
I20260927 00:51:05.373880 18453 consensus_queue.cc:260] T 9954ff2f7f154c0da5c26f4615695d91 P 1dd66d4d49564484bc18587e81d4d9ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c60ef0c03bc4486827edaf45948564f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43023 } } peers { permanent_uuid: "5b39d421768a4a5fb248244a6aebee00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37981 } } peers { permanent_uuid: "1dd66d4d49564484bc18587e81d4d9ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41121 } }
I20260927 00:51:05.373880 18451 consensus_queue.cc:260] T 9954ff2f7f154c0da5c26f4615695d91 P 5b39d421768a4a5fb248244a6aebee00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c60ef0c03bc4486827edaf45948564f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43023 } } peers { permanent_uuid: "5b39d421768a4a5fb248244a6aebee00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37981 } } peers { permanent_uuid: "1dd66d4d49564484bc18587e81d4d9ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41121 } }
I20260927 00:51:05.374418 18452 ts_tablet_manager.cc:1428] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:51:05.374820 18451 ts_tablet_manager.cc:1428] T 9954ff2f7f154c0da5c26f4615695d91 P 5b39d421768a4a5fb248244a6aebee00: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:51:05.374820 18453 ts_tablet_manager.cc:1428] T 9954ff2f7f154c0da5c26f4615695d91 P 1dd66d4d49564484bc18587e81d4d9ad: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:51:05.374871 18303 heartbeater.cc:503] Master 127.0.98.62:38283 was elected leader, sending a full tablet report...
I20260927 00:51:05.375172 18172 heartbeater.cc:503] Master 127.0.98.62:38283 was elected leader, sending a full tablet report...
I20260927 00:51:05.376097 18434 heartbeater.cc:503] Master 127.0.98.62:38283 was elected leader, sending a full tablet report...
I20260927 00:51:05.417557 18457 raft_consensus.cc:482] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:51:05.417797 18457 raft_consensus.cc:504] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c60ef0c03bc4486827edaf45948564f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43023 } } peers { permanent_uuid: "5b39d421768a4a5fb248244a6aebee00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37981 } } peers { permanent_uuid: "1dd66d4d49564484bc18587e81d4d9ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41121 } }
I20260927 00:51:05.419005 18457 leader_election.cc:288] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b39d421768a4a5fb248244a6aebee00 (127.0.98.3:37981), 1dd66d4d49564484bc18587e81d4d9ad (127.0.98.1:41121)
I20260927 00:51:05.423059 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:05.419436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34909 (local address 127.0.98.1:41121)
0927 00:51:05.419584 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:05.419588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:05.419816 (+ 228us) server_negotiation.cc:407] Connection header received
0927 00:51:05.419925 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:05.419928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:05.419981 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:05.420070 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:05.420359 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.420893 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.422006 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.422412 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.422530 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:05.422596 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:05.422835 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 00:51:05.422912 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:51:05.423067 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:05.419436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52761 (local address 127.0.98.3:37981)
0927 00:51:05.419585 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:05.419589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:05.419606 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:51:05.419669 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:05.419672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:05.419724 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:05.419813 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:05.420148 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.420771 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.421984 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.422405 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.422536 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:05.422596 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:05.422871 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 00:51:05.422947 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 00:51:05.423066 18305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:05.419191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37981 (local address 127.0.98.2:52761)
0927 00:51:05.419349 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:05.419366 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:05.419467 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:05.419842 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:05.419845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:05.419882 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:05.420030 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:05.420041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.420882 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.420886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:05.421838 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:05.421847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.422551 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.422554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:05.422717 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:05.422741 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:51:05.422840 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 00:51:05.422918 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":97}
I20260927 00:51:05.423084 18460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:05.419300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41121 (local address 127.0.98.2:34909)
0927 00:51:05.419707 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:05.419721 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:05.419825 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:05.420110 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:05.420114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:05.420132 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:05.420265 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:05.420272 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.421011 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.421014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:05.421838 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:05.421847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.422565 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.422568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:05.422693 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:05.422731 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 00:51:05.422791 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 00:51:05.422891 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":47,"thread_start_us":131,"threads_started":1}
I20260927 00:51:05.423815 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9954ff2f7f154c0da5c26f4615695d91" candidate_uuid: "1c60ef0c03bc4486827edaf45948564f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b39d421768a4a5fb248244a6aebee00" is_pre_election: true
I20260927 00:51:05.424094 18388 raft_consensus.cc:2454] T 9954ff2f7f154c0da5c26f4615695d91 P 5b39d421768a4a5fb248244a6aebee00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c60ef0c03bc4486827edaf45948564f in term 0.
I20260927 00:51:05.424703 18192 leader_election.cc:302] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c60ef0c03bc4486827edaf45948564f, 5b39d421768a4a5fb248244a6aebee00; no voters:
I20260927 00:51:05.425040 18457 raft_consensus.cc:2781] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:51:05.425056 18126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9954ff2f7f154c0da5c26f4615695d91" candidate_uuid: "1c60ef0c03bc4486827edaf45948564f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd66d4d49564484bc18587e81d4d9ad" is_pre_election: true
I20260927 00:51:05.425148 18457 raft_consensus.cc:482] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:51:05.425195 18457 raft_consensus.cc:3037] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:05.425369 18126 raft_consensus.cc:2454] T 9954ff2f7f154c0da5c26f4615695d91 P 1dd66d4d49564484bc18587e81d4d9ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c60ef0c03bc4486827edaf45948564f in term 0.
I20260927 00:51:05.426342 18457 raft_consensus.cc:504] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c60ef0c03bc4486827edaf45948564f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43023 } } peers { permanent_uuid: "5b39d421768a4a5fb248244a6aebee00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37981 } } peers { permanent_uuid: "1dd66d4d49564484bc18587e81d4d9ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41121 } }
I20260927 00:51:05.426702 18457 leader_election.cc:288] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f [CANDIDATE]: Term 1 election: Requested vote from peers 5b39d421768a4a5fb248244a6aebee00 (127.0.98.3:37981), 1dd66d4d49564484bc18587e81d4d9ad (127.0.98.1:41121)
I20260927 00:51:05.427054 18126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9954ff2f7f154c0da5c26f4615695d91" candidate_uuid: "1c60ef0c03bc4486827edaf45948564f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd66d4d49564484bc18587e81d4d9ad"
I20260927 00:51:05.427165 18126 raft_consensus.cc:3037] T 9954ff2f7f154c0da5c26f4615695d91 P 1dd66d4d49564484bc18587e81d4d9ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:05.428306 18126 raft_consensus.cc:2454] T 9954ff2f7f154c0da5c26f4615695d91 P 1dd66d4d49564484bc18587e81d4d9ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c60ef0c03bc4486827edaf45948564f in term 1.
I20260927 00:51:05.428377 18388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9954ff2f7f154c0da5c26f4615695d91" candidate_uuid: "1c60ef0c03bc4486827edaf45948564f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b39d421768a4a5fb248244a6aebee00"
I20260927 00:51:05.428576 18388 raft_consensus.cc:3037] T 9954ff2f7f154c0da5c26f4615695d91 P 5b39d421768a4a5fb248244a6aebee00 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:05.428656 18190 leader_election.cc:302] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c60ef0c03bc4486827edaf45948564f, 1dd66d4d49564484bc18587e81d4d9ad; no voters:
I20260927 00:51:05.428820 18457 raft_consensus.cc:2781] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:05.429083 18457 raft_consensus.cc:686] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f [term 1 LEADER]: Becoming Leader. State: Replica: 1c60ef0c03bc4486827edaf45948564f, State: Running, Role: LEADER
I20260927 00:51:05.429427 18457 consensus_queue.cc:237] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c60ef0c03bc4486827edaf45948564f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43023 } } peers { permanent_uuid: "5b39d421768a4a5fb248244a6aebee00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37981 } } peers { permanent_uuid: "1dd66d4d49564484bc18587e81d4d9ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41121 } }
I20260927 00:51:05.429872 18388 raft_consensus.cc:2454] T 9954ff2f7f154c0da5c26f4615695d91 P 5b39d421768a4a5fb248244a6aebee00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c60ef0c03bc4486827edaf45948564f in term 1.
I20260927 00:51:05.433138 17984 catalog_manager.cc:5495] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c60ef0c03bc4486827edaf45948564f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1c60ef0c03bc4486827edaf45948564f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c60ef0c03bc4486827edaf45948564f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b39d421768a4a5fb248244a6aebee00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dd66d4d49564484bc18587e81d4d9ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41121 } health_report { overall_health: UNKNOWN } } }
I20260927 00:51:05.451900 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:05.448288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38342 (local address 127.0.98.2:43023)
0927 00:51:05.448421 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:05.448424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:05.448438 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:51:05.448541 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:05.448543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:05.448598 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:05.448687 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:51:05.448902 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.449485 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.450305 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:05.450563 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:05.450642 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:05.450985 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:51:05.451445 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:51:05.451553 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:05.451608 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 00:51:05.451661 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:51:05.458312 18126 raft_consensus.cc:1261] T 9954ff2f7f154c0da5c26f4615695d91 P 1dd66d4d49564484bc18587e81d4d9ad [term 1 FOLLOWER]: Refusing update from remote peer 1c60ef0c03bc4486827edaf45948564f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:51:05.458343 18388 raft_consensus.cc:1261] T 9954ff2f7f154c0da5c26f4615695d91 P 5b39d421768a4a5fb248244a6aebee00 [term 1 FOLLOWER]: Refusing update from remote peer 1c60ef0c03bc4486827edaf45948564f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:51:05.459271 18457 consensus_queue.cc:1035] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b39d421768a4a5fb248244a6aebee00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:51:05.459520 18461 consensus_queue.cc:1035] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dd66d4d49564484bc18587e81d4d9ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:51:05.467722 18464 mvcc.cc:205] Tried to move back new op lower bound from 7333766207308849152 to 7333766207202254848. Current Snapshot: MvccSnapshot[applied={T|T < 7333766207308849152}]
I20260927 00:51:05.468124 18466 mvcc.cc:205] Tried to move back new op lower bound from 7333766207308849152 to 7333766207202254848. Current Snapshot: MvccSnapshot[applied={T|T < 7333766207308849152}]
I20260927 00:51:05.477578 17983 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:51:05.477756 17983 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:51:05.480458 17983 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03fb757c36cc459887d5a72927899c3e: Sending DeleteTablet for 3 replicas of tablet 9954ff2f7f154c0da5c26f4615695d91
I20260927 00:51:05.481455 18106 tablet_service.cc:1514] Processing DeleteTablet for tablet 9954ff2f7f154c0da5c26f4615695d91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:51:05 UTC) from {username='slave'} at 127.0.0.1:49582
I20260927 00:51:05.481642 18237 tablet_service.cc:1514] Processing DeleteTablet for tablet 9954ff2f7f154c0da5c26f4615695d91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:51:05 UTC) from {username='slave'} at 127.0.0.1:38326
I20260927 00:51:05.482005 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 18044
I20260927 00:51:05.482187 18368 tablet_service.cc:1514] Processing DeleteTablet for tablet 9954ff2f7f154c0da5c26f4615695d91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:51:05 UTC) from {username='slave'} at 127.0.0.1:49492
I20260927 00:51:05.483384 18476 tablet_replica.cc:330] T 9954ff2f7f154c0da5c26f4615695d91 P 5b39d421768a4a5fb248244a6aebee00: stopping tablet replica
I20260927 00:51:05.483800 18475 tablet_replica.cc:330] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f: stopping tablet replica
I20260927 00:51:05.483925 18476 raft_consensus.cc:2229] T 9954ff2f7f154c0da5c26f4615695d91 P 5b39d421768a4a5fb248244a6aebee00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:51:05.484104 18475 raft_consensus.cc:2229] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:51:05.484401 18476 raft_consensus.cc:2258] T 9954ff2f7f154c0da5c26f4615695d91 P 5b39d421768a4a5fb248244a6aebee00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:51:05.484611 18475 raft_consensus.cc:2258] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:51:05.486093 18475 ts_tablet_manager.cc:1905] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:51:05.486395 18476 ts_tablet_manager.cc:1905] T 9954ff2f7f154c0da5c26f4615695d91 P 5b39d421768a4a5fb248244a6aebee00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:51:05.489167 18475 ts_tablet_manager.cc:1918] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:51:05.489363 18475 log.cc:1196] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-1/wal/wals/9954ff2f7f154c0da5c26f4615695d91
I20260927 00:51:05.489784 18475 ts_tablet_manager.cc:1939] T 9954ff2f7f154c0da5c26f4615695d91 P 1c60ef0c03bc4486827edaf45948564f: Deleting consensus metadata
I20260927 00:51:05.489903 18476 ts_tablet_manager.cc:1918] T 9954ff2f7f154c0da5c26f4615695d91 P 5b39d421768a4a5fb248244a6aebee00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:51:05.490080 18476 log.cc:1196] T 9954ff2f7f154c0da5c26f4615695d91 P 5b39d421768a4a5fb248244a6aebee00: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790470006085796-392-0/minicluster-data/ts-2/wal/wals/9954ff2f7f154c0da5c26f4615695d91
I20260927 00:51:05.490566 18476 ts_tablet_manager.cc:1939] T 9954ff2f7f154c0da5c26f4615695d91 P 5b39d421768a4a5fb248244a6aebee00: Deleting consensus metadata
W20260927 00:51:05.491042 17968 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41121 (error 108)
I20260927 00:51:05.491243 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 18175
I20260927 00:51:05.493394 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:05.493027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41121 (local address 127.0.0.1:49596)
0927 00:51:05.493215 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:05.493295 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260927 00:51:05.493602 17968 catalog_manager.cc:4584] TS 1dd66d4d49564484bc18587e81d4d9ad (127.0.98.1:41121): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9954ff2f7f154c0da5c26f4615695d91: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41121: connect: Connection refused (error 111)
I20260927 00:51:05.493855 17970 catalog_manager.cc:4841] TS 5b39d421768a4a5fb248244a6aebee00 (127.0.98.3:37981): tablet 9954ff2f7f154c0da5c26f4615695d91 (table test-table [id=c082ff91e94141ab9b5ac93dbf6a1aa0]) successfully deleted
I20260927 00:51:05.496903 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 18306
I20260927 00:51:05.497879 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:05.497081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43023 (local address 127.0.0.1:38344)
0927 00:51:05.497700 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:05.497765 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516}
I20260927 00:51:05.503221 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 17953
2026-09-27T00:51:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1131 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:51:05 dist-test-slave-86rs krb5kdc[18480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:51:05 dist-test-slave-86rs krb5kdc[18480](info): set up 2 sockets
Sep 27 00:51:05 dist-test-slave-86rs krb5kdc[18480](info): commencing operation
krb5kdc: starting...
W20260927 00:51:06.548483 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:51:06 dist-test-slave-86rs krb5kdc[18480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:51:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:51:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:51:06.680847 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41667
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:06.795794 18493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:06.796090 18493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:06.796132 18493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:06.800397 18493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:06.800473 18493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:06.800493 18493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:06.800509 18493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:06.800525 18493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:06.806329 18493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:06.807561 18493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:06.808655 18493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:06.817425 18499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:06.817586 18493 server_base.cc:1034] running on GCE node
W20260927 00:51:06.817472 18501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:06.817476 18498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:06.818323 18493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:06.819514 18493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:06.820674 18493 hybrid_clock.cc:648] HybridClock initialized: now 1790470266820661 us; error 26 us; skew 500 ppm
Sep 27 00:51:06 dist-test-slave-86rs krb5kdc[18480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:06.823474 18493 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:51:06.824486 18493 webserver.cc:460] Webserver started at http://127.0.98.62:42229/ using document root <none> and password file <none>
I20260927 00:51:06.825059 18493 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:06.825148 18493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:06.825443 18493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:06.827924 18493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "3aef80356cdf427ea7ae8a0ce0920868"
format_stamp: "Formatted at 2026-09-27 00:51:06 on dist-test-slave-86rs"
server_key: "0eb5aef8ae7e3b59fc430284a3acf340"
server_key_iv: "da46d2a8e0dc5d1a7427fd8c5e312bc1"
server_key_version: "encryptionkey@0"
I20260927 00:51:06.828555 18493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "3aef80356cdf427ea7ae8a0ce0920868"
format_stamp: "Formatted at 2026-09-27 00:51:06 on dist-test-slave-86rs"
server_key: "0eb5aef8ae7e3b59fc430284a3acf340"
server_key_iv: "da46d2a8e0dc5d1a7427fd8c5e312bc1"
server_key_version: "encryptionkey@0"
I20260927 00:51:06.832418 18493 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:51:06.834951 18508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:06.836364 18493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:51:06.836511 18493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "3aef80356cdf427ea7ae8a0ce0920868"
format_stamp: "Formatted at 2026-09-27 00:51:06 on dist-test-slave-86rs"
server_key: "0eb5aef8ae7e3b59fc430284a3acf340"
server_key_iv: "da46d2a8e0dc5d1a7427fd8c5e312bc1"
server_key_version: "encryptionkey@0"
I20260927 00:51:06.836638 18493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:06.853116 18493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:06.856737 18493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:06.856971 18493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:06.864506 18493 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41667
I20260927 00:51:06.864530 18560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41667 every 8 connection(s)
I20260927 00:51:06.865511 18493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:51:06.868168 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 18493
I20260927 00:51:06.868350 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:51:06.868621 18561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:06.868686 392 external_mini_cluster.cc:1428] Setting key 249f84d284541173d66928ae8986d96a
I20260927 00:51:06.875280 18561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aef80356cdf427ea7ae8a0ce0920868: Bootstrap starting.
Sep 27 00:51:06 dist-test-slave-86rs krb5kdc[18480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:51:06.877691 18561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3aef80356cdf427ea7ae8a0ce0920868: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:06.878580 18561 log.cc:824] T 00000000000000000000000000000000 P 3aef80356cdf427ea7ae8a0ce0920868: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:06.880631 18561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aef80356cdf427ea7ae8a0ce0920868: No bootstrap required, opened a new log
I20260927 00:51:06.883136 18564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:06.870154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59410 (local address 127.0.98.62:41667)
0927 00:51:06.870618 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:06.870633 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:06.870667 (+ 34us) server_negotiation.cc:407] Connection header received
0927 00:51:06.871505 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:06.871525 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:06.871854 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:06.872202 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:06.873044 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:06.874062 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:06.874978 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:06.875525 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:06.875690 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:06.878248 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:06.878255 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:06.878271 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:06.878306 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:06.880156 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:06.880723 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:06.880748 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:06.880756 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:06.880892 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:06.881182 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:06.881187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:06.881189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:06.881612 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:06.881815 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:06.882097 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 00:51:06.882404 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":157,"threads_started":1}
I20260927 00:51:06.884411 18561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3aef80356cdf427ea7ae8a0ce0920868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aef80356cdf427ea7ae8a0ce0920868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41667 } }
I20260927 00:51:06.884662 18561 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3aef80356cdf427ea7ae8a0ce0920868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:06.884742 18561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3aef80356cdf427ea7ae8a0ce0920868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aef80356cdf427ea7ae8a0ce0920868, State: Initialized, Role: FOLLOWER
I20260927 00:51:06.885171 18561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3aef80356cdf427ea7ae8a0ce0920868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aef80356cdf427ea7ae8a0ce0920868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41667 } }
I20260927 00:51:06.885339 18561 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3aef80356cdf427ea7ae8a0ce0920868 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:51:06.885411 18561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3aef80356cdf427ea7ae8a0ce0920868 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:51:06.885542 18561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3aef80356cdf427ea7ae8a0ce0920868 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:06.886478 18561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3aef80356cdf427ea7ae8a0ce0920868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aef80356cdf427ea7ae8a0ce0920868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41667 } }
I20260927 00:51:06.886821 18561 leader_election.cc:302] T 00000000000000000000000000000000 P 3aef80356cdf427ea7ae8a0ce0920868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3aef80356cdf427ea7ae8a0ce0920868; no voters:
I20260927 00:51:06.887192 18561 leader_election.cc:288] T 00000000000000000000000000000000 P 3aef80356cdf427ea7ae8a0ce0920868 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:51:06.887343 18566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3aef80356cdf427ea7ae8a0ce0920868 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:06.887557 18566 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3aef80356cdf427ea7ae8a0ce0920868 [term 1 LEADER]: Becoming Leader. State: Replica: 3aef80356cdf427ea7ae8a0ce0920868, State: Running, Role: LEADER
I20260927 00:51:06.887858 18566 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3aef80356cdf427ea7ae8a0ce0920868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aef80356cdf427ea7ae8a0ce0920868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41667 } }
I20260927 00:51:06.888523 18561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3aef80356cdf427ea7ae8a0ce0920868 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:51:06.889103 18567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aef80356cdf427ea7ae8a0ce0920868 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3aef80356cdf427ea7ae8a0ce0920868. Latest consensus state: current_term: 1 leader_uuid: "3aef80356cdf427ea7ae8a0ce0920868" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aef80356cdf427ea7ae8a0ce0920868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41667 } } }
I20260927 00:51:06.889319 18567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aef80356cdf427ea7ae8a0ce0920868 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:06.889544 18568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aef80356cdf427ea7ae8a0ce0920868 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3aef80356cdf427ea7ae8a0ce0920868" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aef80356cdf427ea7ae8a0ce0920868" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41667 } } }
I20260927 00:51:06.889648 18568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aef80356cdf427ea7ae8a0ce0920868 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:06.889791 18574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:51:06.892977 18574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:51:06.899371 18574 catalog_manager.cc:1347] Generated new cluster ID: 4a03fc7783c641b0ac13eec59d81a64d
I20260927 00:51:06.899466 18574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:51:06.932766 18574 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:51:06.933771 18574 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:51:06.942868 18574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3aef80356cdf427ea7ae8a0ce0920868: Generated new TSK 0
I20260927 00:51:06.943641 18574 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:51:06.984119 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41667
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:07.105297 18587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:07.105614 18587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:07.105662 18587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:07.109930 18587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:07.110014 18587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:07.110078 18587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:51:07.115854 18587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:07.117161 18587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:07.118273 18587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:07.128942 18593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:07.128937 18595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:07.129014 18587 server_base.cc:1034] running on GCE node
W20260927 00:51:07.128937 18592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:07.129755 18587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:07.130528 18587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:07.131747 18587 hybrid_clock.cc:648] HybridClock initialized: now 1790470267131716 us; error 57 us; skew 500 ppm
Sep 27 00:51:07 dist-test-slave-86rs krb5kdc[18480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:07.140479 18587 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:51:07.141709 18587 webserver.cc:460] Webserver started at http://127.0.98.1:39549/ using document root <none> and password file <none>
I20260927 00:51:07.142366 18587 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:07.142477 18587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:07.142735 18587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:07.145190 18587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-0/data/instance:
uuid: "0f0e976c46e1400bbb770e67cbf71b20"
format_stamp: "Formatted at 2026-09-27 00:51:07 on dist-test-slave-86rs"
server_key: "f75482907b765c979626ef46d52801bf"
server_key_iv: "333427f19f23ac4f8272466b256ecf7a"
server_key_version: "encryptionkey@0"
I20260927 00:51:07.145840 18587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0f0e976c46e1400bbb770e67cbf71b20"
format_stamp: "Formatted at 2026-09-27 00:51:07 on dist-test-slave-86rs"
server_key: "f75482907b765c979626ef46d52801bf"
server_key_iv: "333427f19f23ac4f8272466b256ecf7a"
server_key_version: "encryptionkey@0"
I20260927 00:51:07.150117 18587 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 00:51:07.153043 18602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:07.154474 18587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:51:07.154646 18587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "0f0e976c46e1400bbb770e67cbf71b20"
format_stamp: "Formatted at 2026-09-27 00:51:07 on dist-test-slave-86rs"
server_key: "f75482907b765c979626ef46d52801bf"
server_key_iv: "333427f19f23ac4f8272466b256ecf7a"
server_key_version: "encryptionkey@0"
I20260927 00:51:07.154788 18587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:07.179653 18587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:07.183055 18587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:07.183281 18587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:07.183946 18587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:51:07.184892 18587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:51:07.184978 18587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:07.185036 18587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:51:07.185088 18587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:07.194774 18587 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33377
I20260927 00:51:07.194796 18715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33377 every 8 connection(s)
I20260927 00:51:07.195739 18587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:51:07.202311 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 18587
I20260927 00:51:07.202487 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:51:07.202795 392 external_mini_cluster.cc:1428] Setting key dd7ea8ba515c76bdbc0cc56cff022b95
Sep 27 00:51:07 dist-test-slave-86rs krb5kdc[18480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:51:07.211548 18564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:07.197745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43419 (local address 127.0.98.62:41667)
0927 00:51:07.197889 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:07.197892 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:07.198758 (+ 866us) server_negotiation.cc:407] Connection header received
0927 00:51:07.200033 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:07.200036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:07.200090 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:07.200175 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:07.201496 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.202148 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.203673 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.204012 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.204134 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:07.206937 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:07.206942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:07.206945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:07.206983 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:07.208928 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:07.209566 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:07.209571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:07.209573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:07.209681 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:07.210145 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:07.210151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:07.210154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:07.210450 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:07.210579 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:07.211181 (+ 602us) server_negotiation.cc:299] Negotiation successful
0927 00:51:07.211340 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:51:07.212559 18718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:07.198086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41667 (local address 127.0.98.1:43419)
0927 00:51:07.198567 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:07.198612 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:07.199792 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:07.200412 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:07.200419 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:07.200821 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:07.201317 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:07.201331 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.202309 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.202312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:07.203543 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:07.203554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.204212 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.204216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:07.204419 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:07.205172 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:51:07.205199 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:07.206727 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:07.209074 (+ 2347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:07.209080 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:07.209105 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:07.209440 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:07.209828 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:07.209831 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:07.209833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:07.210023 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:07.210579 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:07.210585 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:07.210893 (+ 308us) client_negotiation.cc:769] Sending connection context
0927 00:51:07.211088 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 00:51:07.211410 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:51:07.214094 18716 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41667
I20260927 00:51:07.214424 18716 heartbeater.cc:464] Registering TS with master...
I20260927 00:51:07.215009 18716 heartbeater.cc:511] Master 127.0.98.62:41667 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:51:07.217015 18525 ts_manager.cc:194] Registered new tserver with Master: 0f0e976c46e1400bbb770e67cbf71b20 (127.0.98.1:33377)
I20260927 00:51:07.218536 18525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43419
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:51:07.235884 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41667
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:07.354214 18721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:07.354574 18721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:07.354653 18721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:07.358978 18721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:07.359077 18721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:07.359169 18721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:51:07.364833 18721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:07.366015 18721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:07.367072 18721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:07.377166 18729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:07.377140 18726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:07.377140 18727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:07.377466 18721 server_base.cc:1034] running on GCE node
I20260927 00:51:07.378000 18721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:07.378698 18721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:07.379913 18721 hybrid_clock.cc:648] HybridClock initialized: now 1790470267379901 us; error 29 us; skew 500 ppm
Sep 27 00:51:07 dist-test-slave-86rs krb5kdc[18480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:07.383047 18721 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:51:07.384063 18721 webserver.cc:460] Webserver started at http://127.0.98.2:45941/ using document root <none> and password file <none>
I20260927 00:51:07.384742 18721 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:07.384821 18721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:07.385100 18721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:07.387457 18721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-1/data/instance:
uuid: "443e7beb242f473c85247a6f835424a1"
format_stamp: "Formatted at 2026-09-27 00:51:07 on dist-test-slave-86rs"
server_key: "4df113fadc4b27a7eadeb25abc0172fc"
server_key_iv: "1228fd0961fbe6ac6f90f14344199263"
server_key_version: "encryptionkey@0"
I20260927 00:51:07.388129 18721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-1/wal/instance:
uuid: "443e7beb242f473c85247a6f835424a1"
format_stamp: "Formatted at 2026-09-27 00:51:07 on dist-test-slave-86rs"
server_key: "4df113fadc4b27a7eadeb25abc0172fc"
server_key_iv: "1228fd0961fbe6ac6f90f14344199263"
server_key_version: "encryptionkey@0"
I20260927 00:51:07.393076 18721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 00:51:07.396198 18736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:07.397706 18721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:51:07.397825 18721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "443e7beb242f473c85247a6f835424a1"
format_stamp: "Formatted at 2026-09-27 00:51:07 on dist-test-slave-86rs"
server_key: "4df113fadc4b27a7eadeb25abc0172fc"
server_key_iv: "1228fd0961fbe6ac6f90f14344199263"
server_key_version: "encryptionkey@0"
I20260927 00:51:07.397936 18721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:07.412065 18721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:07.415529 18721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:07.415751 18721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:07.416432 18721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:51:07.417318 18721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:51:07.417366 18721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:07.417433 18721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:51:07.417481 18721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:07.427042 18721 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43063
I20260927 00:51:07.427063 18849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43063 every 8 connection(s)
I20260927 00:51:07.428056 18721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:51:07.433014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 18721
I20260927 00:51:07.433159 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:51:07.433460 392 external_mini_cluster.cc:1428] Setting key 67db39d0f6610d8dc0f49870962b58d6
Sep 27 00:51:07 dist-test-slave-86rs krb5kdc[18480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:51:07.443298 18564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:07.430135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51229 (local address 127.0.98.62:41667)
0927 00:51:07.430286 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:07.430290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:07.431158 (+ 868us) server_negotiation.cc:407] Connection header received
0927 00:51:07.432472 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:07.432475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:07.432524 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:07.432605 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:07.434202 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.434751 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.436003 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.436336 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.436465 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:07.439152 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:07.439156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:07.439158 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:07.439184 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:07.440999 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:07.441573 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:07.441577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:07.441578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:07.441665 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:07.442045 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:07.442048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:07.442049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:07.442257 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:07.442365 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:07.442964 (+ 599us) server_negotiation.cc:299] Negotiation successful
0927 00:51:07.443113 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:51:07.444159 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:07.430550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41667 (local address 127.0.98.2:51229)
0927 00:51:07.430994 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:07.431041 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:07.432185 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:07.432835 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:07.432844 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:07.433355 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:07.434039 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:07.434057 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.434893 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.434902 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:07.435850 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:07.435860 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.436530 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.436535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:07.436703 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:07.437378 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:51:07.437395 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:07.438963 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:07.441142 (+ 2179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:07.441149 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:07.441168 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:07.441455 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:07.441761 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:07.441764 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:07.441766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:07.441926 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:07.442365 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:07.442373 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:07.442694 (+ 321us) client_negotiation.cc:769] Sending connection context
0927 00:51:07.442897 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 00:51:07.443257 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":98,"threads_started":1}
I20260927 00:51:07.445590 18850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41667
I20260927 00:51:07.445868 18850 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:51:07.446439 18850 heartbeater.cc:511] Master 127.0.98.62:41667 requested a full tablet report, sending...
I20260927 00:51:07.447745 18525 ts_manager.cc:194] Registered new tserver with Master: 443e7beb242f473c85247a6f835424a1 (127.0.98.2:43063)
I20260927 00:51:07.448665 18525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:51:07.464187 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41667
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:07.579989 18855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:07.580302 18855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:07.580361 18855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:07.584510 18855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:07.584579 18855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:07.584640 18855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:51:07.590302 18855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:07.591523 18855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:07.592681 18855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:07.602914 18860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:07.602985 18855 server_base.cc:1034] running on GCE node
W20260927 00:51:07.602916 18863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:07.602921 18861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:07.603643 18855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:07.604445 18855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:07.605743 18855 hybrid_clock.cc:648] HybridClock initialized: now 1790470267605724 us; error 29 us; skew 500 ppm
Sep 27 00:51:07 dist-test-slave-86rs krb5kdc[18480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:07.609165 18855 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:51:07.610199 18855 webserver.cc:460] Webserver started at http://127.0.98.3:39861/ using document root <none> and password file <none>
I20260927 00:51:07.610798 18855 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:07.610894 18855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:07.611114 18855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:07.613312 18855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-2/data/instance:
uuid: "9ea66cd655a64384b4f7b7784c3a03a8"
format_stamp: "Formatted at 2026-09-27 00:51:07 on dist-test-slave-86rs"
server_key: "9a11170bf662f58fe6f8641ec22c4ed1"
server_key_iv: "917a814d70fe1a0a1e8c37c8ae137f7f"
server_key_version: "encryptionkey@0"
I20260927 00:51:07.613931 18855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9ea66cd655a64384b4f7b7784c3a03a8"
format_stamp: "Formatted at 2026-09-27 00:51:07 on dist-test-slave-86rs"
server_key: "9a11170bf662f58fe6f8641ec22c4ed1"
server_key_iv: "917a814d70fe1a0a1e8c37c8ae137f7f"
server_key_version: "encryptionkey@0"
I20260927 00:51:07.617862 18855 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:51:07.620437 18870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:07.621709 18855 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:51:07.621812 18855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "9ea66cd655a64384b4f7b7784c3a03a8"
format_stamp: "Formatted at 2026-09-27 00:51:07 on dist-test-slave-86rs"
server_key: "9a11170bf662f58fe6f8641ec22c4ed1"
server_key_iv: "917a814d70fe1a0a1e8c37c8ae137f7f"
server_key_version: "encryptionkey@0"
I20260927 00:51:07.621906 18855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:07.657022 18855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:07.660665 18855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:07.660833 18855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:07.661486 18855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:51:07.662338 18855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:51:07.662387 18855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:07.662439 18855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:51:07.662456 18855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:07.671897 18855 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46311
I20260927 00:51:07.671921 18983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46311 every 8 connection(s)
I20260927 00:51:07.672941 18855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:51:07.681658 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 18855
I20260927 00:51:07.681833 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:51:07.682152 392 external_mini_cluster.cc:1428] Setting key b03b3d21dc48dfa5ccd24e34e80664fb
Sep 27 00:51:07 dist-test-slave-86rs krb5kdc[18480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:51:07.687458 18564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:07.674841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39371 (local address 127.0.98.62:41667)
0927 00:51:07.674997 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:07.675001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:07.675853 (+ 852us) server_negotiation.cc:407] Connection header received
0927 00:51:07.677295 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:07.677298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:07.677365 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:07.677484 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:07.678722 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.679277 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.680595 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.680831 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.680947 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:07.683820 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:07.683823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:07.683825 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:07.683854 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:07.685317 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:07.685855 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:07.685859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:07.685861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:07.685953 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:07.686304 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:07.686309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:07.686310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:07.686513 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:07.686620 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:07.687157 (+ 537us) server_negotiation.cc:299] Negotiation successful
0927 00:51:07.687287 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 00:51:07.688387 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:07.675226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41667 (local address 127.0.98.3:39371)
0927 00:51:07.675695 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:07.675739 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:07.677061 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:07.677659 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:07.677666 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:07.678059 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:07.678568 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:07.678582 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.679414 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.679422 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:07.680457 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:07.680468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.681009 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.681013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:07.681192 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:07.681890 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:51:07.681928 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:07.683650 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:07.685447 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:07.685454 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:07.685471 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:07.685744 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:07.686047 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:07.686051 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:07.686053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:07.686204 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:07.686618 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:07.686625 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:07.686905 (+ 280us) client_negotiation.cc:769] Sending connection context
0927 00:51:07.687103 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 00:51:07.687465 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":114,"threads_started":1}
I20260927 00:51:07.689539 18984 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41667
I20260927 00:51:07.689844 18984 heartbeater.cc:464] Registering TS with master...
I20260927 00:51:07.690433 18984 heartbeater.cc:511] Master 127.0.98.62:41667 requested a full tablet report, sending...
I20260927 00:51:07.691643 18525 ts_manager.cc:194] Registered new tserver with Master: 9ea66cd655a64384b4f7b7784c3a03a8 (127.0.98.3:46311)
I20260927 00:51:07.692307 18525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39371
I20260927 00:51:07.696350 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:51:07.706140 18564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:07.698349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59412 (local address 127.0.98.62:41667)
0927 00:51:07.698495 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:07.698498 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:07.698712 (+ 214us) server_negotiation.cc:407] Connection header received
0927 00:51:07.698813 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:07.698815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:07.698854 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:07.698921 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:07.699602 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.700109 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.700977 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.701187 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.701266 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:07.702960 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:07.702963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:07.702965 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:07.702988 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:07.704530 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:07.704982 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:07.704984 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:07.704985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:07.705055 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:07.705311 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:07.705313 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:07.705315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:07.705527 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:07.705673 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:07.705926 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 00:51:07.706007 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:51:07.709447 18525 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:51:07.712667 18525 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:51:07.727890 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:07.721624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33377 (local address 127.0.0.1:43960)
0927 00:51:07.722096 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:07.722144 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:07.722285 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:07.722876 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:07.722882 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:07.722903 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:07.723054 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:07.723062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.724648 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.724654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:07.726145 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:07.726156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.727282 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.727286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:07.727397 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:07.727467 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 00:51:07.727578 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 00:51:07.727713 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":24,"thread_start_us":78,"threads_started":1}
I20260927 00:51:07.727974 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:07.721820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43960 (local address 127.0.98.1:33377)
0927 00:51:07.722365 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:07.722374 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:07.722400 (+ 26us) server_negotiation.cc:407] Connection header received
0927 00:51:07.722493 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:07.722500 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:07.722720 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:07.722878 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:07.723348 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.724533 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.726292 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.727146 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.727261 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:07.727412 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:07.727679 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 00:51:07.727805 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":108,"threads_started":1}
I20260927 00:51:07.728197 18998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:07.722317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46311 (local address 127.0.0.1:37660)
0927 00:51:07.722556 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:07.722568 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:07.722640 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:07.723931 (+ 1291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:07.723940 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:07.723956 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:07.724056 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:07.724062 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.725489 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.725494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:07.726624 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:07.726634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.727686 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.727690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:07.727761 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:07.727782 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:51:07.727845 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 00:51:07.727906 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":1,"thread_start_us":101,"threads_started":1}
I20260927 00:51:07.728588 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:07.721816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43063 (local address 127.0.0.1:56192)
0927 00:51:07.722224 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:07.722240 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:07.722383 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:07.723815 (+ 1432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:07.723818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:07.723836 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:07.723936 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:07.723941 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.725407 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.725411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:07.726677 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:07.726687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.727987 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.727994 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:07.728098 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:07.728120 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:51:07.728189 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 00:51:07.728298 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":121,"threads_started":1}
I20260927 00:51:07.728597 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:07.722718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37660 (local address 127.0.98.3:46311)
0927 00:51:07.723344 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:07.723355 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:07.723375 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:51:07.723456 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:07.723462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:07.723644 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:07.723787 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:07.724272 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.725372 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.726761 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.727387 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.727488 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:07.727629 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:07.728320 (+ 691us) server_negotiation.cc:299] Negotiation successful
0927 00:51:07.728453 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":157,"threads_started":1}
I20260927 00:51:07.728626 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:07.722761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56192 (local address 127.0.98.2:43063)
0927 00:51:07.723201 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:07.723212 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:07.723235 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:51:07.723314 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:07.723319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:07.723503 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:07.723648 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:07.724062 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.725265 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.726818 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.727684 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.727791 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:07.727917 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:07.728382 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 00:51:07.728487 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":125,"threads_started":1}
I20260927 00:51:07.731307 18650 tablet_service.cc:1467] Processing CreateTablet for tablet d2e9a46c8ef8428983e1e41b23285660 (DEFAULT_TABLE table=test-table [id=452a837fe5364878acac7755b222d0c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:51:07.731575 18918 tablet_service.cc:1467] Processing CreateTablet for tablet d2e9a46c8ef8428983e1e41b23285660 (DEFAULT_TABLE table=test-table [id=452a837fe5364878acac7755b222d0c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:51:07.731576 18784 tablet_service.cc:1467] Processing CreateTablet for tablet d2e9a46c8ef8428983e1e41b23285660 (DEFAULT_TABLE table=test-table [id=452a837fe5364878acac7755b222d0c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:51:07.732514 18650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2e9a46c8ef8428983e1e41b23285660. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:07.732652 18784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2e9a46c8ef8428983e1e41b23285660. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:07.732652 18918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2e9a46c8ef8428983e1e41b23285660. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:07.739820 19002 tablet_bootstrap.cc:492] T d2e9a46c8ef8428983e1e41b23285660 P 443e7beb242f473c85247a6f835424a1: Bootstrap starting.
I20260927 00:51:07.740634 19001 tablet_bootstrap.cc:492] T d2e9a46c8ef8428983e1e41b23285660 P 0f0e976c46e1400bbb770e67cbf71b20: Bootstrap starting.
I20260927 00:51:07.742271 19002 tablet_bootstrap.cc:654] T d2e9a46c8ef8428983e1e41b23285660 P 443e7beb242f473c85247a6f835424a1: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:07.742542 19001 tablet_bootstrap.cc:654] T d2e9a46c8ef8428983e1e41b23285660 P 0f0e976c46e1400bbb770e67cbf71b20: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:07.743264 19001 log.cc:824] T d2e9a46c8ef8428983e1e41b23285660 P 0f0e976c46e1400bbb770e67cbf71b20: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:07.743340 19002 log.cc:824] T d2e9a46c8ef8428983e1e41b23285660 P 443e7beb242f473c85247a6f835424a1: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:07.744038 19003 tablet_bootstrap.cc:492] T d2e9a46c8ef8428983e1e41b23285660 P 9ea66cd655a64384b4f7b7784c3a03a8: Bootstrap starting.
I20260927 00:51:07.745473 19002 tablet_bootstrap.cc:492] T d2e9a46c8ef8428983e1e41b23285660 P 443e7beb242f473c85247a6f835424a1: No bootstrap required, opened a new log
I20260927 00:51:07.745496 19001 tablet_bootstrap.cc:492] T d2e9a46c8ef8428983e1e41b23285660 P 0f0e976c46e1400bbb770e67cbf71b20: No bootstrap required, opened a new log
I20260927 00:51:07.745672 19002 ts_tablet_manager.cc:1397] T d2e9a46c8ef8428983e1e41b23285660 P 443e7beb242f473c85247a6f835424a1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:51:07.745700 19001 ts_tablet_manager.cc:1397] T d2e9a46c8ef8428983e1e41b23285660 P 0f0e976c46e1400bbb770e67cbf71b20: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 00:51:07.746280 19003 tablet_bootstrap.cc:654] T d2e9a46c8ef8428983e1e41b23285660 P 9ea66cd655a64384b4f7b7784c3a03a8: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:07.747540 19003 log.cc:824] T d2e9a46c8ef8428983e1e41b23285660 P 9ea66cd655a64384b4f7b7784c3a03a8: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:07.749570 19001 raft_consensus.cc:348] T d2e9a46c8ef8428983e1e41b23285660 P 0f0e976c46e1400bbb770e67cbf71b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0e976c46e1400bbb770e67cbf71b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33377 } } peers { permanent_uuid: "443e7beb242f473c85247a6f835424a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43063 } } peers { permanent_uuid: "9ea66cd655a64384b4f7b7784c3a03a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46311 } }
I20260927 00:51:07.749647 19002 raft_consensus.cc:348] T d2e9a46c8ef8428983e1e41b23285660 P 443e7beb242f473c85247a6f835424a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0e976c46e1400bbb770e67cbf71b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33377 } } peers { permanent_uuid: "443e7beb242f473c85247a6f835424a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43063 } } peers { permanent_uuid: "9ea66cd655a64384b4f7b7784c3a03a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46311 } }
I20260927 00:51:07.749845 19001 raft_consensus.cc:374] T d2e9a46c8ef8428983e1e41b23285660 P 0f0e976c46e1400bbb770e67cbf71b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:07.749861 19003 tablet_bootstrap.cc:492] T d2e9a46c8ef8428983e1e41b23285660 P 9ea66cd655a64384b4f7b7784c3a03a8: No bootstrap required, opened a new log
I20260927 00:51:07.749897 19001 raft_consensus.cc:729] T d2e9a46c8ef8428983e1e41b23285660 P 0f0e976c46e1400bbb770e67cbf71b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0e976c46e1400bbb770e67cbf71b20, State: Initialized, Role: FOLLOWER
I20260927 00:51:07.749912 19002 raft_consensus.cc:374] T d2e9a46c8ef8428983e1e41b23285660 P 443e7beb242f473c85247a6f835424a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:07.749971 19002 raft_consensus.cc:729] T d2e9a46c8ef8428983e1e41b23285660 P 443e7beb242f473c85247a6f835424a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443e7beb242f473c85247a6f835424a1, State: Initialized, Role: FOLLOWER
I20260927 00:51:07.750063 19003 ts_tablet_manager.cc:1397] T d2e9a46c8ef8428983e1e41b23285660 P 9ea66cd655a64384b4f7b7784c3a03a8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:51:07.750384 19001 consensus_queue.cc:260] T d2e9a46c8ef8428983e1e41b23285660 P 0f0e976c46e1400bbb770e67cbf71b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0e976c46e1400bbb770e67cbf71b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33377 } } peers { permanent_uuid: "443e7beb242f473c85247a6f835424a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43063 } } peers { permanent_uuid: "9ea66cd655a64384b4f7b7784c3a03a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46311 } }
I20260927 00:51:07.750428 19002 consensus_queue.cc:260] T d2e9a46c8ef8428983e1e41b23285660 P 443e7beb242f473c85247a6f835424a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0e976c46e1400bbb770e67cbf71b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33377 } } peers { permanent_uuid: "443e7beb242f473c85247a6f835424a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43063 } } peers { permanent_uuid: "9ea66cd655a64384b4f7b7784c3a03a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46311 } }
I20260927 00:51:07.751084 18716 heartbeater.cc:503] Master 127.0.98.62:41667 was elected leader, sending a full tablet report...
I20260927 00:51:07.751453 19002 ts_tablet_manager.cc:1428] T d2e9a46c8ef8428983e1e41b23285660 P 443e7beb242f473c85247a6f835424a1: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260927 00:51:07.751811 18850 heartbeater.cc:503] Master 127.0.98.62:41667 was elected leader, sending a full tablet report...
I20260927 00:51:07.753930 19001 ts_tablet_manager.cc:1428] T d2e9a46c8ef8428983e1e41b23285660 P 0f0e976c46e1400bbb770e67cbf71b20: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260927 00:51:07.753887 19003 raft_consensus.cc:348] T d2e9a46c8ef8428983e1e41b23285660 P 9ea66cd655a64384b4f7b7784c3a03a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0e976c46e1400bbb770e67cbf71b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33377 } } peers { permanent_uuid: "443e7beb242f473c85247a6f835424a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43063 } } peers { permanent_uuid: "9ea66cd655a64384b4f7b7784c3a03a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46311 } }
I20260927 00:51:07.754189 19003 raft_consensus.cc:374] T d2e9a46c8ef8428983e1e41b23285660 P 9ea66cd655a64384b4f7b7784c3a03a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:07.754241 19003 raft_consensus.cc:729] T d2e9a46c8ef8428983e1e41b23285660 P 9ea66cd655a64384b4f7b7784c3a03a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ea66cd655a64384b4f7b7784c3a03a8, State: Initialized, Role: FOLLOWER
I20260927 00:51:07.754743 19003 consensus_queue.cc:260] T d2e9a46c8ef8428983e1e41b23285660 P 9ea66cd655a64384b4f7b7784c3a03a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0e976c46e1400bbb770e67cbf71b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33377 } } peers { permanent_uuid: "443e7beb242f473c85247a6f835424a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43063 } } peers { permanent_uuid: "9ea66cd655a64384b4f7b7784c3a03a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46311 } }
I20260927 00:51:07.755409 18984 heartbeater.cc:503] Master 127.0.98.62:41667 was elected leader, sending a full tablet report...
I20260927 00:51:07.755759 19003 ts_tablet_manager.cc:1428] T d2e9a46c8ef8428983e1e41b23285660 P 9ea66cd655a64384b4f7b7784c3a03a8: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:51:07.757671 19007 raft_consensus.cc:482] T d2e9a46c8ef8428983e1e41b23285660 P 0f0e976c46e1400bbb770e67cbf71b20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:51:07.757843 19007 raft_consensus.cc:504] T d2e9a46c8ef8428983e1e41b23285660 P 0f0e976c46e1400bbb770e67cbf71b20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0e976c46e1400bbb770e67cbf71b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33377 } } peers { permanent_uuid: "443e7beb242f473c85247a6f835424a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43063 } } peers { permanent_uuid: "9ea66cd655a64384b4f7b7784c3a03a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46311 } }
I20260927 00:51:07.759183 19007 leader_election.cc:288] T d2e9a46c8ef8428983e1e41b23285660 P 0f0e976c46e1400bbb770e67cbf71b20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443e7beb242f473c85247a6f835424a1 (127.0.98.2:43063), 9ea66cd655a64384b4f7b7784c3a03a8 (127.0.98.3:46311)
I20260927 00:51:07.763336 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:07.759512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35933 (local address 127.0.98.2:43063)
0927 00:51:07.759649 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:07.759652 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:07.759768 (+ 116us) server_negotiation.cc:407] Connection header received
0927 00:51:07.760027 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:07.760029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:07.760071 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:07.760166 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:07.760559 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.761258 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.762351 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.762782 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.762902 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:07.762977 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:07.763147 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 00:51:07.763218 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:51:07.763353 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:07.759370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43063 (local address 127.0.98.1:35933)
0927 00:51:07.759660 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:07.759677 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:07.759790 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:07.760272 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:07.760275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:07.760288 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:07.760441 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:07.760448 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.761374 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.761379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:07.762229 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:07.762237 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.762905 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.762908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:07.763006 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:07.763029 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:51:07.763122 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 00:51:07.763197 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":132,"threads_started":1}
I20260927 00:51:07.763588 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:07.759872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34291 (local address 127.0.98.3:46311)
0927 00:51:07.759969 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:07.759973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:07.760143 (+ 170us) server_negotiation.cc:407] Connection header received
0927 00:51:07.760207 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:07.760209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:07.760282 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:07.760348 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:07.760839 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.761518 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.762564 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.763013 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.763131 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:07.763194 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:07.763378 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 00:51:07.763458 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 00:51:07.763695 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:07.759787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46311 (local address 127.0.98.1:34291)
0927 00:51:07.760041 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:07.760053 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:07.760144 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:07.760379 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:07.760381 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:07.760397 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:07.760507 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:07.760515 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.761641 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.761645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:07.762432 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:07.762441 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.763134 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.763137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:07.763247 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:07.763271 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:51:07.763361 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 00:51:07.763541 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":3,"thread_start_us":100,"threads_started":1}
I20260927 00:51:07.763980 18804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e9a46c8ef8428983e1e41b23285660" candidate_uuid: "0f0e976c46e1400bbb770e67cbf71b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443e7beb242f473c85247a6f835424a1" is_pre_election: true
I20260927 00:51:07.764367 18804 raft_consensus.cc:2454] T d2e9a46c8ef8428983e1e41b23285660 P 443e7beb242f473c85247a6f835424a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f0e976c46e1400bbb770e67cbf71b20 in term 0.
I20260927 00:51:07.764369 18938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e9a46c8ef8428983e1e41b23285660" candidate_uuid: "0f0e976c46e1400bbb770e67cbf71b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ea66cd655a64384b4f7b7784c3a03a8" is_pre_election: true
I20260927 00:51:07.764631 18938 raft_consensus.cc:2454] T d2e9a46c8ef8428983e1e41b23285660 P 9ea66cd655a64384b4f7b7784c3a03a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f0e976c46e1400bbb770e67cbf71b20 in term 0.
I20260927 00:51:07.764886 18605 leader_election.cc:302] T d2e9a46c8ef8428983e1e41b23285660 P 0f0e976c46e1400bbb770e67cbf71b20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0e976c46e1400bbb770e67cbf71b20, 443e7beb242f473c85247a6f835424a1; no voters:
I20260927 00:51:07.765154 19007 raft_consensus.cc:2781] T d2e9a46c8ef8428983e1e41b23285660 P 0f0e976c46e1400bbb770e67cbf71b20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:51:07.765232 19007 raft_consensus.cc:482] T d2e9a46c8ef8428983e1e41b23285660 P 0f0e976c46e1400bbb770e67cbf71b20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:51:07.765264 19007 raft_consensus.cc:3037] T d2e9a46c8ef8428983e1e41b23285660 P 0f0e976c46e1400bbb770e67cbf71b20 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:07.766377 19007 raft_consensus.cc:504] T d2e9a46c8ef8428983e1e41b23285660 P 0f0e976c46e1400bbb770e67cbf71b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0e976c46e1400bbb770e67cbf71b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33377 } } peers { permanent_uuid: "443e7beb242f473c85247a6f835424a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43063 } } peers { permanent_uuid: "9ea66cd655a64384b4f7b7784c3a03a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46311 } }
I20260927 00:51:07.766726 19007 leader_election.cc:288] T d2e9a46c8ef8428983e1e41b23285660 P 0f0e976c46e1400bbb770e67cbf71b20 [CANDIDATE]: Term 1 election: Requested vote from peers 443e7beb242f473c85247a6f835424a1 (127.0.98.2:43063), 9ea66cd655a64384b4f7b7784c3a03a8 (127.0.98.3:46311)
I20260927 00:51:07.767170 18804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e9a46c8ef8428983e1e41b23285660" candidate_uuid: "0f0e976c46e1400bbb770e67cbf71b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443e7beb242f473c85247a6f835424a1"
I20260927 00:51:07.767211 18938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e9a46c8ef8428983e1e41b23285660" candidate_uuid: "0f0e976c46e1400bbb770e67cbf71b20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ea66cd655a64384b4f7b7784c3a03a8"
I20260927 00:51:07.767304 18804 raft_consensus.cc:3037] T d2e9a46c8ef8428983e1e41b23285660 P 443e7beb242f473c85247a6f835424a1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:07.767406 18938 raft_consensus.cc:3037] T d2e9a46c8ef8428983e1e41b23285660 P 9ea66cd655a64384b4f7b7784c3a03a8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:07.768499 18804 raft_consensus.cc:2454] T d2e9a46c8ef8428983e1e41b23285660 P 443e7beb242f473c85247a6f835424a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f0e976c46e1400bbb770e67cbf71b20 in term 1.
I20260927 00:51:07.768590 18938 raft_consensus.cc:2454] T d2e9a46c8ef8428983e1e41b23285660 P 9ea66cd655a64384b4f7b7784c3a03a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f0e976c46e1400bbb770e67cbf71b20 in term 1.
I20260927 00:51:07.768790 18605 leader_election.cc:302] T d2e9a46c8ef8428983e1e41b23285660 P 0f0e976c46e1400bbb770e67cbf71b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0e976c46e1400bbb770e67cbf71b20, 443e7beb242f473c85247a6f835424a1; no voters:
I20260927 00:51:07.769016 19007 raft_consensus.cc:2781] T d2e9a46c8ef8428983e1e41b23285660 P 0f0e976c46e1400bbb770e67cbf71b20 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:07.769279 19007 raft_consensus.cc:686] T d2e9a46c8ef8428983e1e41b23285660 P 0f0e976c46e1400bbb770e67cbf71b20 [term 1 LEADER]: Becoming Leader. State: Replica: 0f0e976c46e1400bbb770e67cbf71b20, State: Running, Role: LEADER
I20260927 00:51:07.769639 19007 consensus_queue.cc:237] T d2e9a46c8ef8428983e1e41b23285660 P 0f0e976c46e1400bbb770e67cbf71b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0e976c46e1400bbb770e67cbf71b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33377 } } peers { permanent_uuid: "443e7beb242f473c85247a6f835424a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43063 } } peers { permanent_uuid: "9ea66cd655a64384b4f7b7784c3a03a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46311 } }
I20260927 00:51:07.772969 18525 catalog_manager.cc:5495] T d2e9a46c8ef8428983e1e41b23285660 P 0f0e976c46e1400bbb770e67cbf71b20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f0e976c46e1400bbb770e67cbf71b20 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0f0e976c46e1400bbb770e67cbf71b20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f0e976c46e1400bbb770e67cbf71b20" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443e7beb242f473c85247a6f835424a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ea66cd655a64384b4f7b7784c3a03a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46311 } health_report { overall_health: UNKNOWN } } }
I20260927 00:51:07.793869 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:07.789998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43964 (local address 127.0.98.1:33377)
0927 00:51:07.790147 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:07.790154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:07.790172 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:51:07.790341 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:07.790344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:07.790440 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:07.790563 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:51:07.790844 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.791369 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.792283 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:07.792509 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:07.792623 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:07.792948 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:51:07.793383 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:51:07.793542 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:07.793661 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 00:51:07.793703 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:51:07.800549 18804 raft_consensus.cc:1261] T d2e9a46c8ef8428983e1e41b23285660 P 443e7beb242f473c85247a6f835424a1 [term 1 FOLLOWER]: Refusing update from remote peer 0f0e976c46e1400bbb770e67cbf71b20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:51:07.800549 18938 raft_consensus.cc:1261] T d2e9a46c8ef8428983e1e41b23285660 P 9ea66cd655a64384b4f7b7784c3a03a8 [term 1 FOLLOWER]: Refusing update from remote peer 0f0e976c46e1400bbb770e67cbf71b20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:51:07.801358 19012 consensus_queue.cc:1035] T d2e9a46c8ef8428983e1e41b23285660 P 0f0e976c46e1400bbb770e67cbf71b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ea66cd655a64384b4f7b7784c3a03a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:51:07.801597 19007 consensus_queue.cc:1035] T d2e9a46c8ef8428983e1e41b23285660 P 0f0e976c46e1400bbb770e67cbf71b20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "443e7beb242f473c85247a6f835424a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:51:07.822372 18525 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:51:07.822595 18525 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:51:07.825426 18525 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3aef80356cdf427ea7ae8a0ce0920868: Sending DeleteTablet for 3 replicas of tablet d2e9a46c8ef8428983e1e41b23285660
I20260927 00:51:07.826432 18650 tablet_service.cc:1514] Processing DeleteTablet for tablet d2e9a46c8ef8428983e1e41b23285660 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:51:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43960
I20260927 00:51:07.826719 18918 tablet_service.cc:1514] Processing DeleteTablet for tablet d2e9a46c8ef8428983e1e41b23285660 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:51:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37660
I20260927 00:51:07.826990 18784 tablet_service.cc:1514] Processing DeleteTablet for tablet d2e9a46c8ef8428983e1e41b23285660 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:51:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56192
I20260927 00:51:07.827359 19026 tablet_replica.cc:330] T d2e9a46c8ef8428983e1e41b23285660 P 9ea66cd655a64384b4f7b7784c3a03a8: stopping tablet replica
I20260927 00:51:07.827664 19027 tablet_replica.cc:330] T d2e9a46c8ef8428983e1e41b23285660 P 443e7beb242f473c85247a6f835424a1: stopping tablet replica
I20260927 00:51:07.827757 19026 raft_consensus.cc:2229] T d2e9a46c8ef8428983e1e41b23285660 P 9ea66cd655a64384b4f7b7784c3a03a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:51:07.827778 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 18587
I20260927 00:51:07.828110 19027 raft_consensus.cc:2229] T d2e9a46c8ef8428983e1e41b23285660 P 443e7beb242f473c85247a6f835424a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:51:07.828132 19026 raft_consensus.cc:2258] T d2e9a46c8ef8428983e1e41b23285660 P 9ea66cd655a64384b4f7b7784c3a03a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:51:07.828573 19027 raft_consensus.cc:2258] T d2e9a46c8ef8428983e1e41b23285660 P 443e7beb242f473c85247a6f835424a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:51:07.830370 19027 ts_tablet_manager.cc:1905] T d2e9a46c8ef8428983e1e41b23285660 P 443e7beb242f473c85247a6f835424a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:51:07.831380 19026 ts_tablet_manager.cc:1905] T d2e9a46c8ef8428983e1e41b23285660 P 9ea66cd655a64384b4f7b7784c3a03a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:51:07.833330 19027 ts_tablet_manager.cc:1918] T d2e9a46c8ef8428983e1e41b23285660 P 443e7beb242f473c85247a6f835424a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:51:07.833439 19027 log.cc:1196] T d2e9a46c8ef8428983e1e41b23285660 P 443e7beb242f473c85247a6f835424a1: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-1/wal/wals/d2e9a46c8ef8428983e1e41b23285660
I20260927 00:51:07.833765 19027 ts_tablet_manager.cc:1939] T d2e9a46c8ef8428983e1e41b23285660 P 443e7beb242f473c85247a6f835424a1: Deleting consensus metadata
I20260927 00:51:07.834375 19026 ts_tablet_manager.cc:1918] T d2e9a46c8ef8428983e1e41b23285660 P 9ea66cd655a64384b4f7b7784c3a03a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:51:07.834504 19026 log.cc:1196] T d2e9a46c8ef8428983e1e41b23285660 P 9ea66cd655a64384b4f7b7784c3a03a8: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790470006085796-392-0/minicluster-data/ts-2/wal/wals/d2e9a46c8ef8428983e1e41b23285660
I20260927 00:51:07.834857 19026 ts_tablet_manager.cc:1939] T d2e9a46c8ef8428983e1e41b23285660 P 9ea66cd655a64384b4f7b7784c3a03a8: Deleting consensus metadata
I20260927 00:51:07.835276 18511 catalog_manager.cc:4841] TS 443e7beb242f473c85247a6f835424a1 (127.0.98.2:43063): tablet d2e9a46c8ef8428983e1e41b23285660 (table test-table [id=452a837fe5364878acac7755b222d0c7]) successfully deleted
W20260927 00:51:07.835332 18510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33377 (error 108)
I20260927 00:51:07.835487 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 18721
I20260927 00:51:07.836453 18511 catalog_manager.cc:4841] TS 9ea66cd655a64384b4f7b7784c3a03a8 (127.0.98.3:46311): tablet d2e9a46c8ef8428983e1e41b23285660 (table test-table [id=452a837fe5364878acac7755b222d0c7]) successfully deleted
I20260927 00:51:07.837564 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:07.837235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33377 (local address 127.0.0.1:43978)
0927 00:51:07.837423 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:07.837488 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260927 00:51:07.837801 18510 catalog_manager.cc:4584] TS 0f0e976c46e1400bbb770e67cbf71b20 (127.0.98.1:33377): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d2e9a46c8ef8428983e1e41b23285660: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33377: connect: Connection refused (error 111)
I20260927 00:51:07.842628 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 18855
I20260927 00:51:07.848407 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 18493
2026-09-27T00:51:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2346 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T00:51:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:51:07Z Disabled control of system clock
I20260927 00:51:07.879489 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33849
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:51:07.995920 19032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:07.996367 19032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:07.996420 19032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:08.000830 19032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:08.000914 19032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:08.000931 19032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:08.000948 19032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:08.000964 19032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:08.006413 19032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33849
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:08.007592 19032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:08.009127 19032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:08.018246 19040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:08.018246 19038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:08.018251 19037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:08.018662 19032 server_base.cc:1034] running on GCE node
I20260927 00:51:08.019305 19032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:08.020568 19032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:08.021780 19032 hybrid_clock.cc:648] HybridClock initialized: now 1790470268021766 us; error 28 us; skew 500 ppm
I20260927 00:51:08.023828 19032 webserver.cc:460] Webserver started at http://127.0.98.62:45051/ using document root <none> and password file <none>
I20260927 00:51:08.024531 19032 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:08.024627 19032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:08.024875 19032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:08.027105 19032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "82e839749f814751a64ed802beb403d9"
format_stamp: "Formatted at 2026-09-27 00:51:08 on dist-test-slave-86rs"
server_key: "a85cef38865a24d236d4c767386cad30"
server_key_iv: "79145e1506dfb4d0d873bf6a1f1f4739"
server_key_version: "encryptionkey@0"
I20260927 00:51:08.027685 19032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "82e839749f814751a64ed802beb403d9"
format_stamp: "Formatted at 2026-09-27 00:51:08 on dist-test-slave-86rs"
server_key: "a85cef38865a24d236d4c767386cad30"
server_key_iv: "79145e1506dfb4d0d873bf6a1f1f4739"
server_key_version: "encryptionkey@0"
I20260927 00:51:08.031565 19032 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:51:08.034248 19046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:08.035557 19032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:51:08.035717 19032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "82e839749f814751a64ed802beb403d9"
format_stamp: "Formatted at 2026-09-27 00:51:08 on dist-test-slave-86rs"
server_key: "a85cef38865a24d236d4c767386cad30"
server_key_iv: "79145e1506dfb4d0d873bf6a1f1f4739"
server_key_version: "encryptionkey@0"
I20260927 00:51:08.035851 19032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:08.054389 19032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:08.055178 19032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:08.055403 19032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:08.062714 19032 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33849
I20260927 00:51:08.062754 19098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33849 every 8 connection(s)
I20260927 00:51:08.063848 19032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:51:08.066044 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 19032
I20260927 00:51:08.066195 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:51:08.066532 392 external_mini_cluster.cc:1428] Setting key 8276c512ac700ef81cfeed4d1246871a
I20260927 00:51:08.067571 19099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:08.073621 19099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82e839749f814751a64ed802beb403d9: Bootstrap starting.
I20260927 00:51:08.075985 19099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82e839749f814751a64ed802beb403d9: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:08.076135 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:08.068049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34428 (local address 127.0.98.62:33849)
0927 00:51:08.068724 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:08.068739 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:08.068778 (+ 39us) server_negotiation.cc:407] Connection header received
0927 00:51:08.069457 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:08.069473 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:08.069869 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:08.070227 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:08.070715 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:08.071723 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:08.072737 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:08.073232 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:08.073416 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:08.074367 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:08.074372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:08.074386 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:08.074777 (+ 391us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:08.074833 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:08.074852 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:08.074975 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:08.075090 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 00:51:08.075364 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":153,"threads_started":1}
I20260927 00:51:08.076983 19099 log.cc:824] T 00000000000000000000000000000000 P 82e839749f814751a64ed802beb403d9: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:08.079356 19099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82e839749f814751a64ed802beb403d9: No bootstrap required, opened a new log
I20260927 00:51:08.082880 19099 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82e839749f814751a64ed802beb403d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e839749f814751a64ed802beb403d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33849 } }
I20260927 00:51:08.083118 19099 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82e839749f814751a64ed802beb403d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:08.083155 19099 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82e839749f814751a64ed802beb403d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82e839749f814751a64ed802beb403d9, State: Initialized, Role: FOLLOWER
I20260927 00:51:08.083587 19099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82e839749f814751a64ed802beb403d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e839749f814751a64ed802beb403d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33849 } }
I20260927 00:51:08.083722 19099 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82e839749f814751a64ed802beb403d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:51:08.083763 19099 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82e839749f814751a64ed802beb403d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:51:08.083827 19099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82e839749f814751a64ed802beb403d9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:08.084978 19099 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82e839749f814751a64ed802beb403d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e839749f814751a64ed802beb403d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33849 } }
I20260927 00:51:08.085289 19099 leader_election.cc:302] T 00000000000000000000000000000000 P 82e839749f814751a64ed802beb403d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82e839749f814751a64ed802beb403d9; no voters:
I20260927 00:51:08.085584 19099 leader_election.cc:288] T 00000000000000000000000000000000 P 82e839749f814751a64ed802beb403d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:51:08.085709 19104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82e839749f814751a64ed802beb403d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:08.085966 19104 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82e839749f814751a64ed802beb403d9 [term 1 LEADER]: Becoming Leader. State: Replica: 82e839749f814751a64ed802beb403d9, State: Running, Role: LEADER
I20260927 00:51:08.086277 19104 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82e839749f814751a64ed802beb403d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e839749f814751a64ed802beb403d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33849 } }
I20260927 00:51:08.086853 19099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82e839749f814751a64ed802beb403d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:51:08.087983 19105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e839749f814751a64ed802beb403d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82e839749f814751a64ed802beb403d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e839749f814751a64ed802beb403d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33849 } } }
I20260927 00:51:08.088114 19105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e839749f814751a64ed802beb403d9 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:08.088397 19106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e839749f814751a64ed802beb403d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82e839749f814751a64ed802beb403d9. Latest consensus state: current_term: 1 leader_uuid: "82e839749f814751a64ed802beb403d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e839749f814751a64ed802beb403d9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33849 } } }
I20260927 00:51:08.088531 19113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:51:08.088563 19106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e839749f814751a64ed802beb403d9 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:08.092003 19113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:51:08.098263 19113 catalog_manager.cc:1347] Generated new cluster ID: 598f416ae9464f999eddd7179a25d738
I20260927 00:51:08.098387 19113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:51:08.111917 19113 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:51:08.113029 19113 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:51:08.119491 19113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82e839749f814751a64ed802beb403d9: Generated new TSK 0
I20260927 00:51:08.120332 19113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:51:08.124080 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33849
--builtin_ntp_servers=127.0.98.20:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:51:08.263984 19123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:08.264375 19123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:08.264458 19123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:08.268674 19123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:08.268747 19123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:08.268808 19123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:51:08.274039 19123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:08.275274 19123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:08.276832 19123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:08.287147 19128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:08.287169 19129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:08.287151 19131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:08.287436 19123 server_base.cc:1034] running on GCE node
I20260927 00:51:08.288137 19123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:08.288946 19123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:08.290185 19123 hybrid_clock.cc:648] HybridClock initialized: now 1790470268290172 us; error 39 us; skew 500 ppm
I20260927 00:51:08.292411 19123 webserver.cc:460] Webserver started at http://127.0.98.1:35433/ using document root <none> and password file <none>
I20260927 00:51:08.293061 19123 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:08.293157 19123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:08.293423 19123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:08.295730 19123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-0/data/instance:
uuid: "a133a4e9a1c74647b5a23b6a2f53f9f9"
format_stamp: "Formatted at 2026-09-27 00:51:08 on dist-test-slave-86rs"
server_key: "298b44738719ee95258116fef691dffb"
server_key_iv: "87d7aadb4bff310664f6dade6f956404"
server_key_version: "encryptionkey@0"
I20260927 00:51:08.296435 19123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a133a4e9a1c74647b5a23b6a2f53f9f9"
format_stamp: "Formatted at 2026-09-27 00:51:08 on dist-test-slave-86rs"
server_key: "298b44738719ee95258116fef691dffb"
server_key_iv: "87d7aadb4bff310664f6dade6f956404"
server_key_version: "encryptionkey@0"
I20260927 00:51:08.300688 19123 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:51:08.304337 19137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:08.305806 19123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:51:08.305953 19123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "a133a4e9a1c74647b5a23b6a2f53f9f9"
format_stamp: "Formatted at 2026-09-27 00:51:08 on dist-test-slave-86rs"
server_key: "298b44738719ee95258116fef691dffb"
server_key_iv: "87d7aadb4bff310664f6dade6f956404"
server_key_version: "encryptionkey@0"
I20260927 00:51:08.306102 19123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:08.318789 19123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:08.319665 19123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:08.319885 19123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:08.320616 19123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:51:08.321653 19123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:51:08.321710 19123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:08.321782 19123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:51:08.321813 19123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:08.331861 19123 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38937
I20260927 00:51:08.331889 19250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38937 every 8 connection(s)
I20260927 00:51:08.332968 19123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:51:08.341984 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 19123
I20260927 00:51:08.342171 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:51:08.342537 392 external_mini_cluster.cc:1428] Setting key 03a16e59ad33c4bf0fab3cd4dcbbf5d1
I20260927 00:51:08.343401 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:08.334989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52179 (local address 127.0.98.62:33849)
0927 00:51:08.335153 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:08.335157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:08.336420 (+ 1263us) server_negotiation.cc:407] Connection header received
0927 00:51:08.337935 (+ 1515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:08.337938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:08.337995 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:08.338091 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:08.339064 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:08.339738 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:08.341142 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:08.341415 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:08.341505 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:08.342797 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:08.342802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:08.342805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:08.342821 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:08.342847 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:08.342858 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:08.342945 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:08.343165 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 00:51:08.343255 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 00:51:08.344417 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:08.335414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33849 (local address 127.0.98.1:52179)
0927 00:51:08.336204 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:08.336326 (+ 122us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:08.337694 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:08.338271 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:08.338279 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:08.338368 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:08.338905 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:08.338919 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:08.339910 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:08.339914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:08.341010 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:08.341021 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:08.341584 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:08.341587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:08.341752 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:08.342469 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:51:08.342470 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:08.342513 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:51:08.342585 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:08.342981 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:08.342990 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:08.342995 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 00:51:08.343171 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 00:51:08.343492 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":154,"threads_started":1}
I20260927 00:51:08.344885 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33849
--builtin_ntp_servers=127.0.98.20:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:51:08.345779 19251 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33849
I20260927 00:51:08.346035 19251 heartbeater.cc:464] Registering TS with master...
I20260927 00:51:08.346750 19251 heartbeater.cc:511] Master 127.0.98.62:33849 requested a full tablet report, sending...
I20260927 00:51:08.348606 19063 ts_manager.cc:194] Registered new tserver with Master: a133a4e9a1c74647b5a23b6a2f53f9f9 (127.0.98.1:38937)
I20260927 00:51:08.349865 19063 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:52179
W20260927 00:51:08.460090 19254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:08.460445 19254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:08.460487 19254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:08.464689 19254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:08.464763 19254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:08.464823 19254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:51:08.469878 19254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:08.471078 19254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:08.472564 19254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:08.482525 19260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:08.482525 19262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:08.482545 19259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:08.482591 19254 server_base.cc:1034] running on GCE node
I20260927 00:51:08.483234 19254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:08.484016 19254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:08.485179 19254 hybrid_clock.cc:648] HybridClock initialized: now 1790470268485172 us; error 54 us; skew 500 ppm
I20260927 00:51:08.487133 19254 webserver.cc:460] Webserver started at http://127.0.98.2:38791/ using document root <none> and password file <none>
I20260927 00:51:08.487720 19254 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:08.487780 19254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:08.488013 19254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:08.490391 19254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-1/data/instance:
uuid: "265e2a14599b47e0ad40bc9d102a4daf"
format_stamp: "Formatted at 2026-09-27 00:51:08 on dist-test-slave-86rs"
server_key: "f3342437eb5411154d46d19ae854ff7a"
server_key_iv: "61ac1ddc8f8fd6b97d474ee132db1ba2"
server_key_version: "encryptionkey@0"
I20260927 00:51:08.491019 19254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-1/wal/instance:
uuid: "265e2a14599b47e0ad40bc9d102a4daf"
format_stamp: "Formatted at 2026-09-27 00:51:08 on dist-test-slave-86rs"
server_key: "f3342437eb5411154d46d19ae854ff7a"
server_key_iv: "61ac1ddc8f8fd6b97d474ee132db1ba2"
server_key_version: "encryptionkey@0"
I20260927 00:51:08.495091 19254 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 00:51:08.497767 19268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:08.499151 19254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:51:08.499289 19254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "265e2a14599b47e0ad40bc9d102a4daf"
format_stamp: "Formatted at 2026-09-27 00:51:08 on dist-test-slave-86rs"
server_key: "f3342437eb5411154d46d19ae854ff7a"
server_key_iv: "61ac1ddc8f8fd6b97d474ee132db1ba2"
server_key_version: "encryptionkey@0"
I20260927 00:51:08.499418 19254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:08.523146 19254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:08.523902 19254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:08.524112 19254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:08.524768 19254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:51:08.525751 19254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:51:08.525821 19254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:08.525889 19254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:51:08.525939 19254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:08.535068 19254 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44475
I20260927 00:51:08.535084 19381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44475 every 8 connection(s)
I20260927 00:51:08.536039 19254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:51:08.542697 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 19254
I20260927 00:51:08.542881 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:51:08.543218 392 external_mini_cluster.cc:1428] Setting key d91e0e1dc17e3b3f676cfbb0c27ed550
I20260927 00:51:08.545604 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33849
--builtin_ntp_servers=127.0.98.20:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:51:08.546717 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:08.538061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37949 (local address 127.0.98.62:33849)
0927 00:51:08.538226 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:08.538230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:08.539326 (+ 1096us) server_negotiation.cc:407] Connection header received
0927 00:51:08.541164 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:08.541167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:08.541219 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:08.541343 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:08.542321 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:08.542880 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:08.544149 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:08.544476 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:08.544595 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:08.545955 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:08.545963 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:08.545969 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:08.545991 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:08.546022 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:08.546037 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:08.546135 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:08.546381 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 00:51:08.546471 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 00:51:08.547538 19384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:08.538455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33849 (local address 127.0.98.2:37949)
0927 00:51:08.539135 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:08.539179 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:08.540920 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:08.541495 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:08.541503 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:08.541598 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:08.542153 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:08.542168 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:08.543033 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:08.543036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:08.544016 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:08.544026 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:08.544641 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:08.544645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:08.544832 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:08.545595 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:51:08.545597 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:08.545640 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:51:08.545710 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:08.546181 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:08.546189 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:08.546195 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:51:08.546332 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 00:51:08.546592 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":115,"threads_started":1}
I20260927 00:51:08.548952 19382 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33849
I20260927 00:51:08.549216 19382 heartbeater.cc:464] Registering TS with master...
I20260927 00:51:08.549772 19382 heartbeater.cc:511] Master 127.0.98.62:33849 requested a full tablet report, sending...
I20260927 00:51:08.550954 19063 ts_manager.cc:194] Registered new tserver with Master: 265e2a14599b47e0ad40bc9d102a4daf (127.0.98.2:44475)
I20260927 00:51:08.551954 19063 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:37949
W20260927 00:51:08.673197 19385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:08.673508 19385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:08.673553 19385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:08.677927 19385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:08.678004 19385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:08.678061 19385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:51:08.683113 19385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:08.684424 19385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:08.685823 19385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:08.696075 19390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:08.696075 19393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:08.696111 19391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:08.696496 19385 server_base.cc:1034] running on GCE node
I20260927 00:51:08.696947 19385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:08.697688 19385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:08.698892 19385 hybrid_clock.cc:648] HybridClock initialized: now 1790470268698873 us; error 33 us; skew 500 ppm
I20260927 00:51:08.700871 19385 webserver.cc:460] Webserver started at http://127.0.98.3:33187/ using document root <none> and password file <none>
I20260927 00:51:08.701502 19385 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:08.701601 19385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:08.701843 19385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:08.704159 19385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-2/data/instance:
uuid: "f62fa06e0eb9491cbfa9e792ffbb0d44"
format_stamp: "Formatted at 2026-09-27 00:51:08 on dist-test-slave-86rs"
server_key: "882327d68add3315d4863b8715985dc4"
server_key_iv: "c5e475ce2c121d3d444e97f71882ea3d"
server_key_version: "encryptionkey@0"
I20260927 00:51:08.704807 19385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f62fa06e0eb9491cbfa9e792ffbb0d44"
format_stamp: "Formatted at 2026-09-27 00:51:08 on dist-test-slave-86rs"
server_key: "882327d68add3315d4863b8715985dc4"
server_key_iv: "c5e475ce2c121d3d444e97f71882ea3d"
server_key_version: "encryptionkey@0"
I20260927 00:51:08.708825 19385 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:51:08.711495 19399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:08.712857 19385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:51:08.712989 19385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "f62fa06e0eb9491cbfa9e792ffbb0d44"
format_stamp: "Formatted at 2026-09-27 00:51:08 on dist-test-slave-86rs"
server_key: "882327d68add3315d4863b8715985dc4"
server_key_iv: "c5e475ce2c121d3d444e97f71882ea3d"
server_key_version: "encryptionkey@0"
I20260927 00:51:08.713093 19385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:08.738667 19385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:08.739399 19385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:08.739559 19385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:08.740152 19385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:51:08.741191 19385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:51:08.741240 19385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:08.741321 19385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:51:08.741365 19385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:08.750700 19385 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41013
I20260927 00:51:08.750716 19512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41013 every 8 connection(s)
I20260927 00:51:08.751694 19385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:51:08.753784 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 19385
I20260927 00:51:08.753921 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:51:08.754196 392 external_mini_cluster.cc:1428] Setting key a2090dfca0f7193ffeac11ad3fb277ee
I20260927 00:51:08.761998 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:08.753700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51413 (local address 127.0.98.62:33849)
0927 00:51:08.753881 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:08.753888 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:08.755151 (+ 1263us) server_negotiation.cc:407] Connection header received
0927 00:51:08.756726 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:08.756729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:08.756776 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:08.756849 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:08.757832 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:08.758382 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:08.759699 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:08.759947 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:08.760041 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:08.761492 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:08.761495 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:08.761497 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:08.761512 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:08.761532 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:08.761540 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:08.761609 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:08.761801 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 00:51:08.761869 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 00:51:08.762913 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:08.754183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33849 (local address 127.0.98.3:51413)
0927 00:51:08.754992 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:08.755046 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:08.756512 (+ 1466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:08.757036 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:08.757043 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:08.757134 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:08.757664 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:08.757676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:08.758538 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:08.758541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:08.759561 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:08.759572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:08.760198 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:08.760202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:08.760503 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:08.761175 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:51:08.761176 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:08.761220 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:51:08.761291 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:08.761624 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:08.761631 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:08.761636 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 00:51:08.761739 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 00:51:08.761996 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":148,"threads_started":1}
I20260927 00:51:08.764333 19513 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33849
I20260927 00:51:08.764633 19513 heartbeater.cc:464] Registering TS with master...
I20260927 00:51:08.765250 19513 heartbeater.cc:511] Master 127.0.98.62:33849 requested a full tablet report, sending...
I20260927 00:51:08.766476 19063 ts_manager.cc:194] Registered new tserver with Master: f62fa06e0eb9491cbfa9e792ffbb0d44 (127.0.98.3:41013)
I20260927 00:51:08.767266 19063 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:51413
I20260927 00:51:08.768294 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:51:08.774919 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:08.770296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34442 (local address 127.0.98.62:33849)
0927 00:51:08.770437 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:08.770442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:08.770624 (+ 182us) server_negotiation.cc:407] Connection header received
0927 00:51:08.770773 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:08.770776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:08.770827 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:08.770912 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:08.771188 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:08.771838 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:08.773082 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:08.773327 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:08.773434 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:08.774404 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:08.774407 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:08.774410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:08.774425 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:08.774453 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:08.774460 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:08.774590 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:08.774711 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 00:51:08.774750 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:51:08.778213 19063 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:51:08.781319 19063 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:51:08.796590 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:08.791461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60556 (local address 127.0.98.1:38937)
0927 00:51:08.791820 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:08.791833 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:08.791859 (+ 26us) server_negotiation.cc:407] Connection header received
0927 00:51:08.791939 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:08.791947 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:08.792157 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:08.792352 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:08.792689 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:08.793873 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:08.795314 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:08.795858 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:08.795995 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:08.796204 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:08.796323 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 00:51:08.796424 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":122,"threads_started":1}
I20260927 00:51:08.796758 19527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:08.790488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41013 (local address 127.0.0.1:57392)
0927 00:51:08.791219 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:08.791281 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:08.791419 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:08.792608 (+ 1189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:08.792612 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:08.792630 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:08.792718 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:08.792725 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:08.793672 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:08.793678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:08.795111 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:08.795122 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:08.796189 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:08.796193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:08.796326 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:08.796368 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 00:51:08.796429 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 00:51:08.796509 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":44,"spinlock_wait_cycles":15232,"thread_start_us":121,"threads_started":1}
I20260927 00:51:08.796782 19524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:08.790822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44475 (local address 127.0.0.1:51390)
0927 00:51:08.791858 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:08.791872 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:08.791968 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:08.792702 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:08.792705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:08.792723 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:08.792804 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:08.792809 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:08.794255 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:08.794258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:08.795343 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:08.795354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:08.796432 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:08.796437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:08.796516 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:08.796534 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 00:51:08.796577 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 00:51:08.796643 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":969,"mutex_wait_us":4,"thread_start_us":99,"threads_started":1}
I20260927 00:51:08.796591 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:08.790509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38937 (local address 127.0.0.1:60556)
0927 00:51:08.791219 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:08.791277 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:08.791419 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:08.792359 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:08.792365 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:08.792390 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:08.792542 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:08.792549 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:08.793996 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:08.793999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:08.795220 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:08.795230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:08.795988 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:08.795992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:08.796097 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:08.796132 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 00:51:08.796286 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 00:51:08.796416 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":80,"spinlock_wait_cycles":2688,"thread_start_us":94,"threads_started":1}
I20260927 00:51:08.797191 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:08.790604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57392 (local address 127.0.98.3:41013)
0927 00:51:08.792124 (+ 1520us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:08.792137 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:08.792159 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:51:08.792271 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:08.792280 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:08.792464 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:08.792630 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:08.792813 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:08.793555 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:08.795246 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:08.796045 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:08.796669 (+ 624us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:08.796862 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:08.796916 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 00:51:08.797021 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1417,"thread_start_us":90,"threads_started":1}
I20260927 00:51:08.797668 19529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:08.791672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51390 (local address 127.0.98.2:44475)
0927 00:51:08.792204 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:08.792213 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:08.792291 (+ 78us) server_negotiation.cc:407] Connection header received
0927 00:51:08.792367 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:08.792372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:08.792556 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:08.792956 (+ 400us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:08.793093 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:08.794130 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:08.795500 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:08.796304 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:08.797155 (+ 851us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:08.797340 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:08.797390 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 00:51:08.797483 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":122,"threads_started":1}
I20260927 00:51:08.800199 19316 tablet_service.cc:1467] Processing CreateTablet for tablet 5788892bf37f439ebb5a450038f89b9c (DEFAULT_TABLE table=test-table [id=66a064ef462e46e3b89fb44bd5305c20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:51:08.800412 19185 tablet_service.cc:1467] Processing CreateTablet for tablet 5788892bf37f439ebb5a450038f89b9c (DEFAULT_TABLE table=test-table [id=66a064ef462e46e3b89fb44bd5305c20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:51:08.800326 19447 tablet_service.cc:1467] Processing CreateTablet for tablet 5788892bf37f439ebb5a450038f89b9c (DEFAULT_TABLE table=test-table [id=66a064ef462e46e3b89fb44bd5305c20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:51:08.801318 19316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5788892bf37f439ebb5a450038f89b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:08.801337 19447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5788892bf37f439ebb5a450038f89b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:08.801337 19185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5788892bf37f439ebb5a450038f89b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:08.807986 19530 tablet_bootstrap.cc:492] T 5788892bf37f439ebb5a450038f89b9c P 265e2a14599b47e0ad40bc9d102a4daf: Bootstrap starting.
I20260927 00:51:08.809461 19532 tablet_bootstrap.cc:492] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44: Bootstrap starting.
I20260927 00:51:08.810390 19530 tablet_bootstrap.cc:654] T 5788892bf37f439ebb5a450038f89b9c P 265e2a14599b47e0ad40bc9d102a4daf: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:08.811128 19530 log.cc:824] T 5788892bf37f439ebb5a450038f89b9c P 265e2a14599b47e0ad40bc9d102a4daf: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:08.811551 19532 tablet_bootstrap.cc:654] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:08.811813 19531 tablet_bootstrap.cc:492] T 5788892bf37f439ebb5a450038f89b9c P a133a4e9a1c74647b5a23b6a2f53f9f9: Bootstrap starting.
I20260927 00:51:08.812448 19532 log.cc:824] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:08.813357 19530 tablet_bootstrap.cc:492] T 5788892bf37f439ebb5a450038f89b9c P 265e2a14599b47e0ad40bc9d102a4daf: No bootstrap required, opened a new log
I20260927 00:51:08.813553 19530 ts_tablet_manager.cc:1397] T 5788892bf37f439ebb5a450038f89b9c P 265e2a14599b47e0ad40bc9d102a4daf: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:51:08.813890 19531 tablet_bootstrap.cc:654] T 5788892bf37f439ebb5a450038f89b9c P a133a4e9a1c74647b5a23b6a2f53f9f9: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:08.814476 19532 tablet_bootstrap.cc:492] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44: No bootstrap required, opened a new log
I20260927 00:51:08.814600 19531 log.cc:824] T 5788892bf37f439ebb5a450038f89b9c P a133a4e9a1c74647b5a23b6a2f53f9f9: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:08.814661 19532 ts_tablet_manager.cc:1397] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260927 00:51:08.816488 19531 tablet_bootstrap.cc:492] T 5788892bf37f439ebb5a450038f89b9c P a133a4e9a1c74647b5a23b6a2f53f9f9: No bootstrap required, opened a new log
I20260927 00:51:08.816748 19531 ts_tablet_manager.cc:1397] T 5788892bf37f439ebb5a450038f89b9c P a133a4e9a1c74647b5a23b6a2f53f9f9: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:51:08.818274 19530 raft_consensus.cc:348] T 5788892bf37f439ebb5a450038f89b9c P 265e2a14599b47e0ad40bc9d102a4daf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62fa06e0eb9491cbfa9e792ffbb0d44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41013 } } peers { permanent_uuid: "a133a4e9a1c74647b5a23b6a2f53f9f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "265e2a14599b47e0ad40bc9d102a4daf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44475 } }
I20260927 00:51:08.818598 19530 raft_consensus.cc:374] T 5788892bf37f439ebb5a450038f89b9c P 265e2a14599b47e0ad40bc9d102a4daf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:08.818668 19530 raft_consensus.cc:729] T 5788892bf37f439ebb5a450038f89b9c P 265e2a14599b47e0ad40bc9d102a4daf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 265e2a14599b47e0ad40bc9d102a4daf, State: Initialized, Role: FOLLOWER
I20260927 00:51:08.819036 19532 raft_consensus.cc:348] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62fa06e0eb9491cbfa9e792ffbb0d44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41013 } } peers { permanent_uuid: "a133a4e9a1c74647b5a23b6a2f53f9f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "265e2a14599b47e0ad40bc9d102a4daf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44475 } }
I20260927 00:51:08.819155 19530 consensus_queue.cc:260] T 5788892bf37f439ebb5a450038f89b9c P 265e2a14599b47e0ad40bc9d102a4daf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62fa06e0eb9491cbfa9e792ffbb0d44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41013 } } peers { permanent_uuid: "a133a4e9a1c74647b5a23b6a2f53f9f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "265e2a14599b47e0ad40bc9d102a4daf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44475 } }
I20260927 00:51:08.819284 19532 raft_consensus.cc:374] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:08.819337 19532 raft_consensus.cc:729] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f62fa06e0eb9491cbfa9e792ffbb0d44, State: Initialized, Role: FOLLOWER
I20260927 00:51:08.819793 19532 consensus_queue.cc:260] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62fa06e0eb9491cbfa9e792ffbb0d44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41013 } } peers { permanent_uuid: "a133a4e9a1c74647b5a23b6a2f53f9f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "265e2a14599b47e0ad40bc9d102a4daf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44475 } }
I20260927 00:51:08.820310 19530 ts_tablet_manager.cc:1428] T 5788892bf37f439ebb5a450038f89b9c P 265e2a14599b47e0ad40bc9d102a4daf: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 00:51:08.820493 19531 raft_consensus.cc:348] T 5788892bf37f439ebb5a450038f89b9c P a133a4e9a1c74647b5a23b6a2f53f9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62fa06e0eb9491cbfa9e792ffbb0d44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41013 } } peers { permanent_uuid: "a133a4e9a1c74647b5a23b6a2f53f9f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "265e2a14599b47e0ad40bc9d102a4daf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44475 } }
I20260927 00:51:08.820683 19382 heartbeater.cc:503] Master 127.0.98.62:33849 was elected leader, sending a full tablet report...
I20260927 00:51:08.820739 19532 ts_tablet_manager.cc:1428] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260927 00:51:08.820747 19531 raft_consensus.cc:374] T 5788892bf37f439ebb5a450038f89b9c P a133a4e9a1c74647b5a23b6a2f53f9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:08.820822 19531 raft_consensus.cc:729] T 5788892bf37f439ebb5a450038f89b9c P a133a4e9a1c74647b5a23b6a2f53f9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a133a4e9a1c74647b5a23b6a2f53f9f9, State: Initialized, Role: FOLLOWER
I20260927 00:51:08.821054 19513 heartbeater.cc:503] Master 127.0.98.62:33849 was elected leader, sending a full tablet report...
I20260927 00:51:08.821271 19531 consensus_queue.cc:260] T 5788892bf37f439ebb5a450038f89b9c P a133a4e9a1c74647b5a23b6a2f53f9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62fa06e0eb9491cbfa9e792ffbb0d44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41013 } } peers { permanent_uuid: "a133a4e9a1c74647b5a23b6a2f53f9f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "265e2a14599b47e0ad40bc9d102a4daf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44475 } }
I20260927 00:51:08.821942 19251 heartbeater.cc:503] Master 127.0.98.62:33849 was elected leader, sending a full tablet report...
I20260927 00:51:08.823465 19531 ts_tablet_manager.cc:1428] T 5788892bf37f439ebb5a450038f89b9c P a133a4e9a1c74647b5a23b6a2f53f9f9: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260927 00:51:08.834210 19252 tablet.cc:2369] T 5788892bf37f439ebb5a450038f89b9c P a133a4e9a1c74647b5a23b6a2f53f9f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:51:09.002761 19514 tablet.cc:2369] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:51:09.037377 19383 tablet.cc:2369] T 5788892bf37f439ebb5a450038f89b9c P 265e2a14599b47e0ad40bc9d102a4daf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:51:09.041390 19537 raft_consensus.cc:482] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:51:09.041628 19537 raft_consensus.cc:504] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62fa06e0eb9491cbfa9e792ffbb0d44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41013 } } peers { permanent_uuid: "a133a4e9a1c74647b5a23b6a2f53f9f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "265e2a14599b47e0ad40bc9d102a4daf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44475 } }
I20260927 00:51:09.042881 19537 leader_election.cc:288] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a133a4e9a1c74647b5a23b6a2f53f9f9 (127.0.98.1:38937), 265e2a14599b47e0ad40bc9d102a4daf (127.0.98.2:44475)
I20260927 00:51:09.046590 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:09.043131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44475 (local address 127.0.98.3:54517)
0927 00:51:09.043250 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:09.043265 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:09.043370 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:09.043756 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:09.043760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:09.043783 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:09.043910 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:09.043919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:09.044723 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:09.044727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:09.045511 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:09.045520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:09.046165 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:09.046169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:09.046270 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:09.046294 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:51:09.046381 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 00:51:09.046452 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 00:51:09.046590 19529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:09.043281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54517 (local address 127.0.98.2:44475)
0927 00:51:09.043435 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:09.043462 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:09.043478 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:51:09.043537 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:09.043540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:09.043600 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:09.043686 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:09.044068 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:09.044601 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:09.045637 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:09.046039 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:09.046159 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:09.046223 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:09.046397 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 00:51:09.046474 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 00:51:09.047118 19539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:09.043138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38937 (local address 127.0.98.3:47183)
0927 00:51:09.043891 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:09.043906 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:09.044019 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:09.044348 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:09.044351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:09.044368 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:09.044501 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:09.044506 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:09.045220 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:09.045225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:09.045979 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:09.045988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:09.046644 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:09.046648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:09.046756 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:09.046779 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:51:09.046891 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 00:51:09.046955 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":321,"thread_start_us":150,"threads_started":1}
I20260927 00:51:09.047142 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:09.043265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47183 (local address 127.0.98.1:38937)
0927 00:51:09.043424 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:09.043428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:09.044007 (+ 579us) server_negotiation.cc:407] Connection header received
0927 00:51:09.044154 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:09.044156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:09.044201 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:09.044328 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:09.044623 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:09.045105 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:09.046111 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:09.046509 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:09.046632 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:09.046689 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:09.046945 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 00:51:09.047030 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 00:51:09.047304 19336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5788892bf37f439ebb5a450038f89b9c" candidate_uuid: "f62fa06e0eb9491cbfa9e792ffbb0d44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "265e2a14599b47e0ad40bc9d102a4daf" is_pre_election: true
I20260927 00:51:09.047674 19336 raft_consensus.cc:2454] T 5788892bf37f439ebb5a450038f89b9c P 265e2a14599b47e0ad40bc9d102a4daf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f62fa06e0eb9491cbfa9e792ffbb0d44 in term 0.
I20260927 00:51:09.047784 19203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5788892bf37f439ebb5a450038f89b9c" candidate_uuid: "f62fa06e0eb9491cbfa9e792ffbb0d44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a133a4e9a1c74647b5a23b6a2f53f9f9" is_pre_election: true
I20260927 00:51:09.048117 19203 raft_consensus.cc:2454] T 5788892bf37f439ebb5a450038f89b9c P a133a4e9a1c74647b5a23b6a2f53f9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f62fa06e0eb9491cbfa9e792ffbb0d44 in term 0.
I20260927 00:51:09.048257 19402 leader_election.cc:302] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 265e2a14599b47e0ad40bc9d102a4daf, f62fa06e0eb9491cbfa9e792ffbb0d44; no voters:
I20260927 00:51:09.048530 19537 raft_consensus.cc:2781] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:51:09.048614 19537 raft_consensus.cc:482] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:51:09.048681 19537 raft_consensus.cc:3037] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:09.049705 19537 raft_consensus.cc:504] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62fa06e0eb9491cbfa9e792ffbb0d44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41013 } } peers { permanent_uuid: "a133a4e9a1c74647b5a23b6a2f53f9f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "265e2a14599b47e0ad40bc9d102a4daf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44475 } }
I20260927 00:51:09.050088 19537 leader_election.cc:288] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44 [CANDIDATE]: Term 1 election: Requested vote from peers a133a4e9a1c74647b5a23b6a2f53f9f9 (127.0.98.1:38937), 265e2a14599b47e0ad40bc9d102a4daf (127.0.98.2:44475)
I20260927 00:51:09.050420 19336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5788892bf37f439ebb5a450038f89b9c" candidate_uuid: "f62fa06e0eb9491cbfa9e792ffbb0d44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "265e2a14599b47e0ad40bc9d102a4daf"
I20260927 00:51:09.050419 19203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5788892bf37f439ebb5a450038f89b9c" candidate_uuid: "f62fa06e0eb9491cbfa9e792ffbb0d44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a133a4e9a1c74647b5a23b6a2f53f9f9"
I20260927 00:51:09.050549 19336 raft_consensus.cc:3037] T 5788892bf37f439ebb5a450038f89b9c P 265e2a14599b47e0ad40bc9d102a4daf [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:09.050549 19203 raft_consensus.cc:3037] T 5788892bf37f439ebb5a450038f89b9c P a133a4e9a1c74647b5a23b6a2f53f9f9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:09.051810 19203 raft_consensus.cc:2454] T 5788892bf37f439ebb5a450038f89b9c P a133a4e9a1c74647b5a23b6a2f53f9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f62fa06e0eb9491cbfa9e792ffbb0d44 in term 1.
I20260927 00:51:09.051810 19336 raft_consensus.cc:2454] T 5788892bf37f439ebb5a450038f89b9c P 265e2a14599b47e0ad40bc9d102a4daf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f62fa06e0eb9491cbfa9e792ffbb0d44 in term 1.
I20260927 00:51:09.052102 19402 leader_election.cc:302] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 265e2a14599b47e0ad40bc9d102a4daf, f62fa06e0eb9491cbfa9e792ffbb0d44; no voters:
I20260927 00:51:09.052294 19537 raft_consensus.cc:2781] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:09.052529 19537 raft_consensus.cc:686] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44 [term 1 LEADER]: Becoming Leader. State: Replica: f62fa06e0eb9491cbfa9e792ffbb0d44, State: Running, Role: LEADER
I20260927 00:51:09.052898 19537 consensus_queue.cc:237] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62fa06e0eb9491cbfa9e792ffbb0d44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41013 } } peers { permanent_uuid: "a133a4e9a1c74647b5a23b6a2f53f9f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } } peers { permanent_uuid: "265e2a14599b47e0ad40bc9d102a4daf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44475 } }
I20260927 00:51:09.056157 19063 catalog_manager.cc:5495] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44 reported cstate change: term changed from 0 to 1, leader changed from <none> to f62fa06e0eb9491cbfa9e792ffbb0d44 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f62fa06e0eb9491cbfa9e792ffbb0d44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62fa06e0eb9491cbfa9e792ffbb0d44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a133a4e9a1c74647b5a23b6a2f53f9f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "265e2a14599b47e0ad40bc9d102a4daf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44475 } health_report { overall_health: UNKNOWN } } }
I20260927 00:51:09.093396 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:09.089592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57398 (local address 127.0.98.3:41013)
0927 00:51:09.089726 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:09.089730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:09.089742 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:51:09.089816 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:09.089818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:09.089876 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:09.089974 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:51:09.090198 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:09.090716 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:09.091674 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:09.091900 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:09.091988 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:09.092426 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:51:09.092865 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:51:09.092979 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:09.093180 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 00:51:09.093216 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:51:09.099965 19203 raft_consensus.cc:1261] T 5788892bf37f439ebb5a450038f89b9c P a133a4e9a1c74647b5a23b6a2f53f9f9 [term 1 FOLLOWER]: Refusing update from remote peer f62fa06e0eb9491cbfa9e792ffbb0d44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:51:09.099959 19336 raft_consensus.cc:1261] T 5788892bf37f439ebb5a450038f89b9c P 265e2a14599b47e0ad40bc9d102a4daf [term 1 FOLLOWER]: Refusing update from remote peer f62fa06e0eb9491cbfa9e792ffbb0d44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:51:09.100805 19540 consensus_queue.cc:1035] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a133a4e9a1c74647b5a23b6a2f53f9f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:51:09.101033 19537 consensus_queue.cc:1035] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "265e2a14599b47e0ad40bc9d102a4daf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:51:09.110627 19543 mvcc.cc:205] Tried to move back new op lower bound from 7333766222224646144 to 7333766222043557888. Current Snapshot: MvccSnapshot[applied={T|T < 7333766222224646144}]
I20260927 00:51:09.118904 19063 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:51:09.119102 19063 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:51:09.121692 19063 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82e839749f814751a64ed802beb403d9: Sending DeleteTablet for 3 replicas of tablet 5788892bf37f439ebb5a450038f89b9c
I20260927 00:51:09.122583 19447 tablet_service.cc:1514] Processing DeleteTablet for tablet 5788892bf37f439ebb5a450038f89b9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:51:09 UTC) from {username='slave'} at 127.0.0.1:57392
I20260927 00:51:09.122581 19185 tablet_service.cc:1514] Processing DeleteTablet for tablet 5788892bf37f439ebb5a450038f89b9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:51:09 UTC) from {username='slave'} at 127.0.0.1:60556
I20260927 00:51:09.122576 19316 tablet_service.cc:1514] Processing DeleteTablet for tablet 5788892bf37f439ebb5a450038f89b9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:51:09 UTC) from {username='slave'} at 127.0.0.1:51390
I20260927 00:51:09.123147 19554 tablet_replica.cc:330] T 5788892bf37f439ebb5a450038f89b9c P 265e2a14599b47e0ad40bc9d102a4daf: stopping tablet replica
I20260927 00:51:09.123292 19552 tablet_replica.cc:330] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44: stopping tablet replica
I20260927 00:51:09.123574 19552 raft_consensus.cc:2229] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:51:09.123924 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 19123
I20260927 00:51:09.124027 19552 raft_consensus.cc:2258] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:51:09.126859 19552 ts_tablet_manager.cc:1905] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:51:09.128540 19554 raft_consensus.cc:2229] T 5788892bf37f439ebb5a450038f89b9c P 265e2a14599b47e0ad40bc9d102a4daf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:51:09.128975 19554 raft_consensus.cc:2258] T 5788892bf37f439ebb5a450038f89b9c P 265e2a14599b47e0ad40bc9d102a4daf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:51:09.130599 19552 ts_tablet_manager.cc:1918] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:51:09.130714 19552 log.cc:1196] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790470006085796-392-0/minicluster-data/ts-2/wal/wals/5788892bf37f439ebb5a450038f89b9c
I20260927 00:51:09.130847 19554 ts_tablet_manager.cc:1905] T 5788892bf37f439ebb5a450038f89b9c P 265e2a14599b47e0ad40bc9d102a4daf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:51:09.131067 19552 ts_tablet_manager.cc:1939] T 5788892bf37f439ebb5a450038f89b9c P f62fa06e0eb9491cbfa9e792ffbb0d44: Deleting consensus metadata
I20260927 00:51:09.132012 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 19254
W20260927 00:51:09.132022 19050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38937 (error 108)
I20260927 00:51:09.137765 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:09.137389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38937 (local address 127.0.0.1:60572)
0927 00:51:09.137589 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:09.137661 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260927 00:51:09.137775 19048 catalog_manager.cc:4841] TS f62fa06e0eb9491cbfa9e792ffbb0d44 (127.0.98.3:41013): tablet 5788892bf37f439ebb5a450038f89b9c (table test-table [id=66a064ef462e46e3b89fb44bd5305c20]) successfully deleted
W20260927 00:51:09.138113 19050 catalog_manager.cc:4584] TS a133a4e9a1c74647b5a23b6a2f53f9f9 (127.0.98.1:38937): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5788892bf37f439ebb5a450038f89b9c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38937: connect: Connection refused (error 111)
W20260927 00:51:09.138504 19525 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:51:09.138067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44475 (local address 127.0.0.1:51400)
0927 00:51:09.138188 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:09.138202 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:09.138290 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:09.138436 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44475: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 00:51:09.138724 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 19385
I20260927 00:51:09.144703 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 19032
2026-09-27T00:51:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1295 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:51:09 dist-test-slave-86rs krb5kdc[19558](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:51:09 dist-test-slave-86rs krb5kdc[19558](info): set up 2 sockets
Sep 27 00:51:09 dist-test-slave-86rs krb5kdc[19558](info): commencing operation
krb5kdc: starting...
W20260927 00:51:10.180514 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:51:10 dist-test-slave-86rs krb5kdc[19558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:51:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:51:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:51:10.309561 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46353
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46353
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:10.428992 19571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:10.429314 19571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:10.429394 19571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:10.433626 19571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:10.433706 19571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:10.433727 19571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:10.433744 19571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:10.433840 19571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:10.439473 19571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:10.440699 19571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:10.441776 19571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:10.450491 19577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:10.450551 19571 server_base.cc:1034] running on GCE node
W20260927 00:51:10.450623 19579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:10.450491 19576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:10.451436 19571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:10.452744 19571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:10.453944 19571 hybrid_clock.cc:648] HybridClock initialized: now 1790470270453915 us; error 43 us; skew 500 ppm
Sep 27 00:51:10 dist-test-slave-86rs krb5kdc[19558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:10.456832 19571 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:51:10.457778 19571 webserver.cc:460] Webserver started at http://127.0.98.62:40635/ using document root <none> and password file <none>
I20260927 00:51:10.458387 19571 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:10.458478 19571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:10.458717 19571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:10.460894 19571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "9fdd4f20e57c4e2db17d8bbffd04e5c2"
format_stamp: "Formatted at 2026-09-27 00:51:10 on dist-test-slave-86rs"
server_key: "a3335d225b3ff44d4d7b7ee2fc55fdf9"
server_key_iv: "2849156b597b3241fdc22f78b43b65b3"
server_key_version: "encryptionkey@0"
I20260927 00:51:10.461472 19571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "9fdd4f20e57c4e2db17d8bbffd04e5c2"
format_stamp: "Formatted at 2026-09-27 00:51:10 on dist-test-slave-86rs"
server_key: "a3335d225b3ff44d4d7b7ee2fc55fdf9"
server_key_iv: "2849156b597b3241fdc22f78b43b65b3"
server_key_version: "encryptionkey@0"
I20260927 00:51:10.465256 19571 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:51:10.467685 19586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:10.468968 19571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:51:10.469117 19571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "9fdd4f20e57c4e2db17d8bbffd04e5c2"
format_stamp: "Formatted at 2026-09-27 00:51:10 on dist-test-slave-86rs"
server_key: "a3335d225b3ff44d4d7b7ee2fc55fdf9"
server_key_iv: "2849156b597b3241fdc22f78b43b65b3"
server_key_version: "encryptionkey@0"
I20260927 00:51:10.469246 19571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:10.482897 19571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:10.486114 19571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:10.486342 19571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:10.493696 19571 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46353
I20260927 00:51:10.493722 19638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46353 every 8 connection(s)
I20260927 00:51:10.494717 19571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:51:10.497673 19639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:10.503527 19639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fdd4f20e57c4e2db17d8bbffd04e5c2: Bootstrap starting.
I20260927 00:51:10.503798 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 19571
I20260927 00:51:10.503966 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:51:10.504290 392 external_mini_cluster.cc:1428] Setting key 891977087115de67675154c8d67fd7d3
I20260927 00:51:10.506477 19639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fdd4f20e57c4e2db17d8bbffd04e5c2: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:10.507277 19639 log.cc:824] T 00000000000000000000000000000000 P 9fdd4f20e57c4e2db17d8bbffd04e5c2: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:10.509332 19639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fdd4f20e57c4e2db17d8bbffd04e5c2: No bootstrap required, opened a new log
I20260927 00:51:10.513270 19639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fdd4f20e57c4e2db17d8bbffd04e5c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fdd4f20e57c4e2db17d8bbffd04e5c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46353 } }
I20260927 00:51:10.513604 19639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fdd4f20e57c4e2db17d8bbffd04e5c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:10.513659 19639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fdd4f20e57c4e2db17d8bbffd04e5c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fdd4f20e57c4e2db17d8bbffd04e5c2, State: Initialized, Role: FOLLOWER
Sep 27 00:51:10 dist-test-slave-86rs krb5kdc[19558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:51:10.514171 19639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fdd4f20e57c4e2db17d8bbffd04e5c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fdd4f20e57c4e2db17d8bbffd04e5c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46353 } }
I20260927 00:51:10.514338 19639 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fdd4f20e57c4e2db17d8bbffd04e5c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:51:10.514420 19639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fdd4f20e57c4e2db17d8bbffd04e5c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:51:10.514557 19639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fdd4f20e57c4e2db17d8bbffd04e5c2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:10.515767 19639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fdd4f20e57c4e2db17d8bbffd04e5c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fdd4f20e57c4e2db17d8bbffd04e5c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46353 } }
I20260927 00:51:10.516132 19639 leader_election.cc:302] T 00000000000000000000000000000000 P 9fdd4f20e57c4e2db17d8bbffd04e5c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fdd4f20e57c4e2db17d8bbffd04e5c2; no voters:
I20260927 00:51:10.516611 19639 leader_election.cc:288] T 00000000000000000000000000000000 P 9fdd4f20e57c4e2db17d8bbffd04e5c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:51:10.516726 19644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fdd4f20e57c4e2db17d8bbffd04e5c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:10.516945 19644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fdd4f20e57c4e2db17d8bbffd04e5c2 [term 1 LEADER]: Becoming Leader. State: Replica: 9fdd4f20e57c4e2db17d8bbffd04e5c2, State: Running, Role: LEADER
I20260927 00:51:10.517290 19644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fdd4f20e57c4e2db17d8bbffd04e5c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fdd4f20e57c4e2db17d8bbffd04e5c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46353 } }
I20260927 00:51:10.518005 19639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fdd4f20e57c4e2db17d8bbffd04e5c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:51:10.519201 19645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fdd4f20e57c4e2db17d8bbffd04e5c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fdd4f20e57c4e2db17d8bbffd04e5c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fdd4f20e57c4e2db17d8bbffd04e5c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46353 } } }
I20260927 00:51:10.519356 19645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fdd4f20e57c4e2db17d8bbffd04e5c2 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:10.519711 19653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:51:10.519770 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:10.505766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41394 (local address 127.0.98.62:46353)
0927 00:51:10.506261 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:10.506280 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:10.506320 (+ 40us) server_negotiation.cc:407] Connection header received
0927 00:51:10.507088 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:10.507108 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:10.507504 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:10.507870 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:10.508731 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:10.509813 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:10.510698 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:10.511319 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:10.511490 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:10.514331 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:10.514338 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:10.514357 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:10.514392 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:10.516308 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:10.516887 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:10.516895 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:10.516906 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:10.517124 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:10.517402 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:10.517405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:10.517407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:10.517916 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:10.518109 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:10.518409 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 00:51:10.518774 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":165,"threads_started":1}
I20260927 00:51:10.520242 19646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fdd4f20e57c4e2db17d8bbffd04e5c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fdd4f20e57c4e2db17d8bbffd04e5c2. Latest consensus state: current_term: 1 leader_uuid: "9fdd4f20e57c4e2db17d8bbffd04e5c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fdd4f20e57c4e2db17d8bbffd04e5c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46353 } } }
I20260927 00:51:10.520336 19646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fdd4f20e57c4e2db17d8bbffd04e5c2 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:10.523005 19653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:51:10.529304 19653 catalog_manager.cc:1347] Generated new cluster ID: 05be666d4e0e478191d0e748cba68385
I20260927 00:51:10.529399 19653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:51:10.538494 19653 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:51:10.539810 19653 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:51:10.546581 19653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fdd4f20e57c4e2db17d8bbffd04e5c2: Generated new TSK 0
I20260927 00:51:10.547374 19653 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:51:10.584550 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46353
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:10.708352 19665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:10.708657 19665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:10.708705 19665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:10.713269 19665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:10.713364 19665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:10.713493 19665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:51:10.719586 19665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:10.720782 19665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:10.721854 19665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:10.731822 19673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:10.731824 19671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:10.731824 19670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:10.732196 19665 server_base.cc:1034] running on GCE node
I20260927 00:51:10.732651 19665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:10.733371 19665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:10.734577 19665 hybrid_clock.cc:648] HybridClock initialized: now 1790470270734550 us; error 41 us; skew 500 ppm
Sep 27 00:51:10 dist-test-slave-86rs krb5kdc[19558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:10.737650 19665 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:51:10.738705 19665 webserver.cc:460] Webserver started at http://127.0.98.1:37563/ using document root <none> and password file <none>
I20260927 00:51:10.739303 19665 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:10.739400 19665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:10.739645 19665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:10.741976 19665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-0/data/instance:
uuid: "47861012027e456f9dfec8bdb6ca88fd"
format_stamp: "Formatted at 2026-09-27 00:51:10 on dist-test-slave-86rs"
server_key: "34479a9a0b76599e4da0c903fa56d9b5"
server_key_iv: "36ea9c1be098f78778f84bef811e4134"
server_key_version: "encryptionkey@0"
I20260927 00:51:10.742576 19665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-0/wal/instance:
uuid: "47861012027e456f9dfec8bdb6ca88fd"
format_stamp: "Formatted at 2026-09-27 00:51:10 on dist-test-slave-86rs"
server_key: "34479a9a0b76599e4da0c903fa56d9b5"
server_key_iv: "36ea9c1be098f78778f84bef811e4134"
server_key_version: "encryptionkey@0"
I20260927 00:51:10.746654 19665 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:51:10.749444 19680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:10.750742 19665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:51:10.750916 19665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "47861012027e456f9dfec8bdb6ca88fd"
format_stamp: "Formatted at 2026-09-27 00:51:10 on dist-test-slave-86rs"
server_key: "34479a9a0b76599e4da0c903fa56d9b5"
server_key_iv: "36ea9c1be098f78778f84bef811e4134"
server_key_version: "encryptionkey@0"
I20260927 00:51:10.751056 19665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:10.791482 19665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:10.794807 19665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:10.795054 19665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:10.795697 19665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:51:10.796661 19665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:51:10.796736 19665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:10.796815 19665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:51:10.796859 19665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:10.806246 19665 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42791
I20260927 00:51:10.806268 19793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42791 every 8 connection(s)
I20260927 00:51:10.807263 19665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:51:10.812691 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 19665
I20260927 00:51:10.812929 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:51:10.813341 392 external_mini_cluster.cc:1428] Setting key 1e6db0b0215c73b4678ae329d07cf39f
Sep 27 00:51:10 dist-test-slave-86rs krb5kdc[19558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:51:10.824095 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:10.809292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36791 (local address 127.0.98.62:46353)
0927 00:51:10.809445 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:10.809449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:10.810285 (+ 836us) server_negotiation.cc:407] Connection header received
0927 00:51:10.811641 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:10.811645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:10.811707 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:10.811789 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:10.813642 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:10.814293 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:10.815801 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:10.816283 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:10.816414 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:10.819876 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:10.819880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:10.819882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:10.819909 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:10.821502 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:10.822201 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:10.822204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:10.822205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:10.822282 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:10.822688 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:10.822690 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:10.822692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:10.822876 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:10.822951 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:10.823755 (+ 804us) server_negotiation.cc:299] Negotiation successful
0927 00:51:10.823871 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 00:51:10.825084 19796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:10.809682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46353 (local address 127.0.98.1:36791)
0927 00:51:10.810127 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:10.810171 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:10.811407 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:10.811982 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:10.811990 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:10.812524 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:10.813438 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:10.813473 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:10.814442 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:10.814446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:10.815645 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:10.815656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:10.816459 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:10.816463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:10.816627 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:10.817357 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:51:10.817384 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:10.819662 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:10.821647 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:10.821655 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:10.821677 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:10.822091 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:10.822376 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:10.822379 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:10.822381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:10.822591 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:10.822999 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:10.823008 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:10.823464 (+ 456us) client_negotiation.cc:769] Sending connection context
0927 00:51:10.823716 (+ 252us) client_negotiation.cc:240] Negotiation successful
0927 00:51:10.824134 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":118,"threads_started":1}
I20260927 00:51:10.826495 19794 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:51:10.826799 19794 heartbeater.cc:464] Registering TS with master...
I20260927 00:51:10.827384 19794 heartbeater.cc:511] Master 127.0.98.62:46353 requested a full tablet report, sending...
I20260927 00:51:10.829101 19603 ts_manager.cc:194] Registered new tserver with Master: 47861012027e456f9dfec8bdb6ca88fd (127.0.98.1:42791)
I20260927 00:51:10.830293 19603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:51:10.847776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46353
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:10.963593 19799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:10.963914 19799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:10.963963 19799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:10.968287 19799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:10.968361 19799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:10.968426 19799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:51:10.973968 19799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:10.975065 19799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:10.976065 19799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:10.986313 19805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:10.986267 19804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:10.986413 19799 server_base.cc:1034] running on GCE node
W20260927 00:51:10.986495 19807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:10.986963 19799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:10.987658 19799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:10.988831 19799 hybrid_clock.cc:648] HybridClock initialized: now 1790470270988811 us; error 29 us; skew 500 ppm
Sep 27 00:51:10 dist-test-slave-86rs krb5kdc[19558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:10.991753 19799 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:51:10.992712 19799 webserver.cc:460] Webserver started at http://127.0.98.2:33931/ using document root <none> and password file <none>
I20260927 00:51:10.993259 19799 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:10.993319 19799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:10.993494 19799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:10.995563 19799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-1/data/instance:
uuid: "47cb6a6aaa4845088804eb9b81414e43"
format_stamp: "Formatted at 2026-09-27 00:51:10 on dist-test-slave-86rs"
server_key: "c821252f2dedb161036d90fe394c586b"
server_key_iv: "f226f4ea16ff0fb9b757947974288066"
server_key_version: "encryptionkey@0"
I20260927 00:51:10.996095 19799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-1/wal/instance:
uuid: "47cb6a6aaa4845088804eb9b81414e43"
format_stamp: "Formatted at 2026-09-27 00:51:10 on dist-test-slave-86rs"
server_key: "c821252f2dedb161036d90fe394c586b"
server_key_iv: "f226f4ea16ff0fb9b757947974288066"
server_key_version: "encryptionkey@0"
I20260927 00:51:11.000095 19799 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 00:51:11.002867 19814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:11.004390 19799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:51:11.004542 19799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "47cb6a6aaa4845088804eb9b81414e43"
format_stamp: "Formatted at 2026-09-27 00:51:10 on dist-test-slave-86rs"
server_key: "c821252f2dedb161036d90fe394c586b"
server_key_iv: "f226f4ea16ff0fb9b757947974288066"
server_key_version: "encryptionkey@0"
I20260927 00:51:11.004678 19799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:11.028167 19799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:11.031829 19799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:11.032056 19799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:11.032820 19799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:51:11.033725 19799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:51:11.033810 19799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:11.033905 19799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:51:11.033957 19799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:11.043056 19799 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34665
I20260927 00:51:11.043079 19927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34665 every 8 connection(s)
I20260927 00:51:11.044073 19799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:51:11.045204 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 19799
I20260927 00:51:11.045372 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:51:11.045681 392 external_mini_cluster.cc:1428] Setting key e20b0f0507c79b4b2947bad413667241
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 27 00:51:11 dist-test-slave-86rs krb5kdc[19558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:51:11.061106 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:11.046415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35125 (local address 127.0.98.62:46353)
0927 00:51:11.046555 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:11.046560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:11.047631 (+ 1071us) server_negotiation.cc:407] Connection header received
0927 00:51:11.049630 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:11.049636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:11.049702 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:11.049806 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:11.051031 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.051954 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.053329 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.053727 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.053860 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:11.056927 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:11.056931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:11.056933 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:11.056961 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:11.058632 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:11.059325 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:11.059329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:11.059330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:11.059430 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:11.059856 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:11.059860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:11.059861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:11.060064 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:11.060180 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:11.060782 (+ 602us) server_negotiation.cc:299] Negotiation successful
0927 00:51:11.060929 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:51:11.061991 19930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:11.046880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46353 (local address 127.0.98.2:35125)
0927 00:51:11.047457 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:11.047518 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:11.049384 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:11.049979 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:11.049986 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:11.050362 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:11.050868 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:11.050882 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.052116 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.052119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:11.053175 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:11.053187 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.053930 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.053934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:11.054163 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:11.054889 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:51:11.054917 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:11.056720 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:11.058749 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:11.058754 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:11.058771 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:11.059127 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:11.059542 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:11.059545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:11.059547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:11.059716 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:11.060178 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:11.060186 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:11.060518 (+ 332us) client_negotiation.cc:769] Sending connection context
0927 00:51:11.060721 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 00:51:11.061093 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":155,"threads_started":1}
I20260927 00:51:11.063551 19928 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46353
I20260927 00:51:11.063858 19928 heartbeater.cc:464] Registering TS with master...
I20260927 00:51:11.064574 19928 heartbeater.cc:511] Master 127.0.98.62:46353 requested a full tablet report, sending...
I20260927 00:51:11.066159 19603 ts_manager.cc:194] Registered new tserver with Master: 47cb6a6aaa4845088804eb9b81414e43 (127.0.98.2:34665)
I20260927 00:51:11.067530 19603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:51:11.077431 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46353
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:11.194841 19933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:11.195134 19933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:11.195182 19933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:11.199433 19933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:11.199510 19933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:11.199625 19933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:51:11.205300 19933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:11.206447 19933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:11.207507 19933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:11.217598 19938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:11.217630 19939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:11.217747 19933 server_base.cc:1034] running on GCE node
W20260927 00:51:11.217672 19941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:11.218405 19933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:11.219235 19933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:11.220396 19933 hybrid_clock.cc:648] HybridClock initialized: now 1790470271220350 us; error 59 us; skew 500 ppm
Sep 27 00:51:11 dist-test-slave-86rs krb5kdc[19558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:11.223346 19933 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:51:11.224334 19933 webserver.cc:460] Webserver started at http://127.0.98.3:36475/ using document root <none> and password file <none>
I20260927 00:51:11.224927 19933 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:11.225028 19933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:11.225266 19933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:11.227360 19933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-2/data/instance:
uuid: "2cd15a8b0ca745608ac5bccb1f3a3fd6"
format_stamp: "Formatted at 2026-09-27 00:51:11 on dist-test-slave-86rs"
server_key: "1c060e0c8b334222c7ce087b782c6060"
server_key_iv: "a0ba67f0ea6a3dd5eafcacf9d5c82afe"
server_key_version: "encryptionkey@0"
I20260927 00:51:11.227922 19933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2cd15a8b0ca745608ac5bccb1f3a3fd6"
format_stamp: "Formatted at 2026-09-27 00:51:11 on dist-test-slave-86rs"
server_key: "1c060e0c8b334222c7ce087b782c6060"
server_key_iv: "a0ba67f0ea6a3dd5eafcacf9d5c82afe"
server_key_version: "encryptionkey@0"
I20260927 00:51:11.231709 19933 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:51:11.234382 19948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:11.235757 19933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 00:51:11.235937 19933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "2cd15a8b0ca745608ac5bccb1f3a3fd6"
format_stamp: "Formatted at 2026-09-27 00:51:11 on dist-test-slave-86rs"
server_key: "1c060e0c8b334222c7ce087b782c6060"
server_key_iv: "a0ba67f0ea6a3dd5eafcacf9d5c82afe"
server_key_version: "encryptionkey@0"
I20260927 00:51:11.236081 19933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:11.263846 19933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:11.267620 19933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:11.267835 19933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:11.268532 19933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:51:11.269399 19933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:51:11.269472 19933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:11.269546 19933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:51:11.269596 19933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:11.278715 19933 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44971
I20260927 00:51:11.278731 20061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44971 every 8 connection(s)
I20260927 00:51:11.279644 19933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:51:11.284925 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 19933
I20260927 00:51:11.285082 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:51:11.285406 392 external_mini_cluster.cc:1428] Setting key 362c2426a1196808ede4225152064a4a
Sep 27 00:51:11 dist-test-slave-86rs krb5kdc[19558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:51:11.293540 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:11.281530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34109 (local address 127.0.98.62:46353)
0927 00:51:11.281716 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:11.281721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:11.282581 (+ 860us) server_negotiation.cc:407] Connection header received
0927 00:51:11.283851 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:11.283854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:11.283905 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:11.283983 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:11.285653 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.286185 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.287379 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.287617 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.287731 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:11.290262 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:11.290266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:11.290268 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:11.290322 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:11.291596 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:11.292073 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:11.292076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:11.292077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:11.292148 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:11.292523 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:11.292525 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:11.292527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:11.292713 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:11.292792 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:11.293279 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 00:51:11.293411 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 00:51:11.294472 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:11.281908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46353 (local address 127.0.98.3:34109)
0927 00:51:11.282416 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:11.282460 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:11.283636 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:11.284163 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:11.284172 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:11.284756 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:11.285483 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:11.285505 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.286324 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.286327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:11.287249 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:11.287259 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.287773 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.287777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:11.287966 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:11.288632 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:51:11.288649 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:11.290096 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:11.291698 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:11.291703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:11.291718 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:11.291985 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:11.292269 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:11.292272 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:11.292274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:11.292420 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:11.292801 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:11.292806 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:11.293062 (+ 256us) client_negotiation.cc:769] Sending connection context
0927 00:51:11.293218 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 00:51:11.293517 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":128,"threads_started":1}
I20260927 00:51:11.295770 20062 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46353
I20260927 00:51:11.296037 20062 heartbeater.cc:464] Registering TS with master...
I20260927 00:51:11.296633 20062 heartbeater.cc:511] Master 127.0.98.62:46353 requested a full tablet report, sending...
I20260927 00:51:11.297885 19603 ts_manager.cc:194] Registered new tserver with Master: 2cd15a8b0ca745608ac5bccb1f3a3fd6 (127.0.98.3:44971)
I20260927 00:51:11.298622 19603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34109
I20260927 00:51:11.299396 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:51:11.308540 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:11.301490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43178 (local address 127.0.98.62:46353)
0927 00:51:11.301611 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:11.301614 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:11.301783 (+ 169us) server_negotiation.cc:407] Connection header received
0927 00:51:11.301875 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:11.301877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:11.301913 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:11.301983 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:11.302628 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.303127 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.303922 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.304126 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.304201 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:11.305631 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:11.305634 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:11.305636 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:11.305654 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:11.306869 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:11.307251 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:11.307253 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:11.307254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:11.307330 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:11.307556 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:11.307558 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:11.307559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:11.307773 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:11.307873 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:11.308097 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 00:51:11.308364 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:51:11.311772 19603 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:51:11.315235 19603 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:51:11.331714 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:11.325547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42791 (local address 127.0.0.1:57016)
0927 00:51:11.326281 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:11.326295 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:11.326384 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:11.327100 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:11.327105 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:11.327127 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:11.327245 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:11.327251 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.328342 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.328346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:11.330087 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:11.330097 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.331141 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.331145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:11.331257 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:11.331297 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 00:51:11.331420 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 00:51:11.331554 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":121,"threads_started":1}
I20260927 00:51:11.331727 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:11.326149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57016 (local address 127.0.98.1:42791)
0927 00:51:11.326644 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:11.326652 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:11.326675 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:51:11.326762 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:11.326766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:11.326934 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:11.327053 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:11.327389 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.328161 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.330248 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.331004 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.331140 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:11.331320 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:11.331452 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 00:51:11.331554 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":125,"threads_started":1}
I20260927 00:51:11.332207 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:11.325106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34665 (local address 127.0.0.1:43654)
0927 00:51:11.325960 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:11.326013 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:11.326149 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:11.327908 (+ 1759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:11.327911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:11.327929 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:11.328043 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:11.328051 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.329603 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.329606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:11.330534 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:11.330545 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.331815 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.331818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:11.331908 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:11.331929 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:51:11.331999 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 00:51:11.332067 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":84,"threads_started":1}
I20260927 00:51:11.333385 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:11.325874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44971 (local address 127.0.0.1:33930)
0927 00:51:11.327238 (+ 1364us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:11.327259 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:11.327333 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:11.328838 (+ 1505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:11.328842 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:11.328864 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:11.328961 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:11.328968 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.330449 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.330453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:11.331619 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:11.331628 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.332920 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.332924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:11.333045 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:11.333067 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:51:11.333170 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 00:51:11.333242 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1311,"mutex_wait_us":1,"thread_start_us":98,"threads_started":1}
I20260927 00:51:11.333463 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:11.326030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33930 (local address 127.0.98.3:44971)
0927 00:51:11.326472 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:11.326484 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:11.328364 (+ 1880us) server_negotiation.cc:407] Connection header received
0927 00:51:11.328474 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:11.328479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:11.328681 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:11.329113 (+ 432us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:11.329280 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.330323 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.331749 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.332630 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.332729 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:11.332860 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:11.333204 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 00:51:11.333302 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":138,"threads_started":1}
I20260927 00:51:11.333637 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:11.325326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43654 (local address 127.0.98.2:34665)
0927 00:51:11.327542 (+ 2216us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:11.327550 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:11.327569 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:51:11.327629 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:11.327632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:11.327769 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:11.327925 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:11.328137 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.329450 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.330669 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.331499 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.331605 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:11.331758 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:11.332358 (+ 600us) server_negotiation.cc:299] Negotiation successful
0927 00:51:11.333480 (+ 1122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2125,"thread_start_us":166,"threads_started":1}
I20260927 00:51:11.335496 19728 tablet_service.cc:1467] Processing CreateTablet for tablet 23a68638c57e4002aa5cbc7d915dd3de (DEFAULT_TABLE table=test-table [id=3daea2e536a044488c34591a8dbc98ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:51:11.336128 19862 tablet_service.cc:1467] Processing CreateTablet for tablet 23a68638c57e4002aa5cbc7d915dd3de (DEFAULT_TABLE table=test-table [id=3daea2e536a044488c34591a8dbc98ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:51:11.336128 19996 tablet_service.cc:1467] Processing CreateTablet for tablet 23a68638c57e4002aa5cbc7d915dd3de (DEFAULT_TABLE table=test-table [id=3daea2e536a044488c34591a8dbc98ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:51:11.337224 19728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23a68638c57e4002aa5cbc7d915dd3de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:11.337236 19862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23a68638c57e4002aa5cbc7d915dd3de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:11.337235 19996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23a68638c57e4002aa5cbc7d915dd3de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:11.344192 20079 tablet_bootstrap.cc:492] T 23a68638c57e4002aa5cbc7d915dd3de P 47861012027e456f9dfec8bdb6ca88fd: Bootstrap starting.
I20260927 00:51:11.345239 20081 tablet_bootstrap.cc:492] T 23a68638c57e4002aa5cbc7d915dd3de P 2cd15a8b0ca745608ac5bccb1f3a3fd6: Bootstrap starting.
I20260927 00:51:11.346532 20079 tablet_bootstrap.cc:654] T 23a68638c57e4002aa5cbc7d915dd3de P 47861012027e456f9dfec8bdb6ca88fd: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:11.347015 20081 tablet_bootstrap.cc:654] T 23a68638c57e4002aa5cbc7d915dd3de P 2cd15a8b0ca745608ac5bccb1f3a3fd6: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:11.347601 20079 log.cc:824] T 23a68638c57e4002aa5cbc7d915dd3de P 47861012027e456f9dfec8bdb6ca88fd: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:11.348270 20081 log.cc:824] T 23a68638c57e4002aa5cbc7d915dd3de P 2cd15a8b0ca745608ac5bccb1f3a3fd6: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:11.348414 20080 tablet_bootstrap.cc:492] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43: Bootstrap starting.
I20260927 00:51:11.349769 20079 tablet_bootstrap.cc:492] T 23a68638c57e4002aa5cbc7d915dd3de P 47861012027e456f9dfec8bdb6ca88fd: No bootstrap required, opened a new log
I20260927 00:51:11.349969 20079 ts_tablet_manager.cc:1397] T 23a68638c57e4002aa5cbc7d915dd3de P 47861012027e456f9dfec8bdb6ca88fd: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 00:51:11.350214 20081 tablet_bootstrap.cc:492] T 23a68638c57e4002aa5cbc7d915dd3de P 2cd15a8b0ca745608ac5bccb1f3a3fd6: No bootstrap required, opened a new log
I20260927 00:51:11.350384 20081 ts_tablet_manager.cc:1397] T 23a68638c57e4002aa5cbc7d915dd3de P 2cd15a8b0ca745608ac5bccb1f3a3fd6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:51:11.350735 20080 tablet_bootstrap.cc:654] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:11.351393 20080 log.cc:824] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:11.353075 20080 tablet_bootstrap.cc:492] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43: No bootstrap required, opened a new log
I20260927 00:51:11.353282 20080 ts_tablet_manager.cc:1397] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:51:11.354347 20079 raft_consensus.cc:348] T 23a68638c57e4002aa5cbc7d915dd3de P 47861012027e456f9dfec8bdb6ca88fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cb6a6aaa4845088804eb9b81414e43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34665 } } peers { permanent_uuid: "2cd15a8b0ca745608ac5bccb1f3a3fd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44971 } } peers { permanent_uuid: "47861012027e456f9dfec8bdb6ca88fd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42791 } }
I20260927 00:51:11.354347 20081 raft_consensus.cc:348] T 23a68638c57e4002aa5cbc7d915dd3de P 2cd15a8b0ca745608ac5bccb1f3a3fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cb6a6aaa4845088804eb9b81414e43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34665 } } peers { permanent_uuid: "2cd15a8b0ca745608ac5bccb1f3a3fd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44971 } } peers { permanent_uuid: "47861012027e456f9dfec8bdb6ca88fd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42791 } }
I20260927 00:51:11.354624 20079 raft_consensus.cc:374] T 23a68638c57e4002aa5cbc7d915dd3de P 47861012027e456f9dfec8bdb6ca88fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:11.354645 20081 raft_consensus.cc:374] T 23a68638c57e4002aa5cbc7d915dd3de P 2cd15a8b0ca745608ac5bccb1f3a3fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:11.354691 20081 raft_consensus.cc:729] T 23a68638c57e4002aa5cbc7d915dd3de P 2cd15a8b0ca745608ac5bccb1f3a3fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd15a8b0ca745608ac5bccb1f3a3fd6, State: Initialized, Role: FOLLOWER
I20260927 00:51:11.354687 20079 raft_consensus.cc:729] T 23a68638c57e4002aa5cbc7d915dd3de P 47861012027e456f9dfec8bdb6ca88fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47861012027e456f9dfec8bdb6ca88fd, State: Initialized, Role: FOLLOWER
I20260927 00:51:11.355142 20081 consensus_queue.cc:260] T 23a68638c57e4002aa5cbc7d915dd3de P 2cd15a8b0ca745608ac5bccb1f3a3fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cb6a6aaa4845088804eb9b81414e43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34665 } } peers { permanent_uuid: "2cd15a8b0ca745608ac5bccb1f3a3fd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44971 } } peers { permanent_uuid: "47861012027e456f9dfec8bdb6ca88fd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42791 } }
I20260927 00:51:11.355142 20079 consensus_queue.cc:260] T 23a68638c57e4002aa5cbc7d915dd3de P 47861012027e456f9dfec8bdb6ca88fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cb6a6aaa4845088804eb9b81414e43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34665 } } peers { permanent_uuid: "2cd15a8b0ca745608ac5bccb1f3a3fd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44971 } } peers { permanent_uuid: "47861012027e456f9dfec8bdb6ca88fd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42791 } }
I20260927 00:51:11.356133 20079 ts_tablet_manager.cc:1428] T 23a68638c57e4002aa5cbc7d915dd3de P 47861012027e456f9dfec8bdb6ca88fd: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260927 00:51:11.356133 20081 ts_tablet_manager.cc:1428] T 23a68638c57e4002aa5cbc7d915dd3de P 2cd15a8b0ca745608ac5bccb1f3a3fd6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:51:11.356562 19794 heartbeater.cc:503] Master 127.0.98.62:46353 was elected leader, sending a full tablet report...
I20260927 00:51:11.356885 20080 raft_consensus.cc:348] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cb6a6aaa4845088804eb9b81414e43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34665 } } peers { permanent_uuid: "2cd15a8b0ca745608ac5bccb1f3a3fd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44971 } } peers { permanent_uuid: "47861012027e456f9dfec8bdb6ca88fd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42791 } }
I20260927 00:51:11.357151 20080 raft_consensus.cc:374] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:11.357213 20080 raft_consensus.cc:729] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47cb6a6aaa4845088804eb9b81414e43, State: Initialized, Role: FOLLOWER
I20260927 00:51:11.357645 20062 heartbeater.cc:503] Master 127.0.98.62:46353 was elected leader, sending a full tablet report...
I20260927 00:51:11.357640 20080 consensus_queue.cc:260] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cb6a6aaa4845088804eb9b81414e43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34665 } } peers { permanent_uuid: "2cd15a8b0ca745608ac5bccb1f3a3fd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44971 } } peers { permanent_uuid: "47861012027e456f9dfec8bdb6ca88fd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42791 } }
I20260927 00:51:11.358290 19928 heartbeater.cc:503] Master 127.0.98.62:46353 was elected leader, sending a full tablet report...
I20260927 00:51:11.358901 20080 ts_tablet_manager.cc:1428] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260927 00:51:11.530658 20063 tablet.cc:2369] T 23a68638c57e4002aa5cbc7d915dd3de P 2cd15a8b0ca745608ac5bccb1f3a3fd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:51:11.545557 19929 tablet.cc:2369] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:51:11.558728 19795 tablet.cc:2369] T 23a68638c57e4002aa5cbc7d915dd3de P 47861012027e456f9dfec8bdb6ca88fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:51:11.595574 20087 raft_consensus.cc:482] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:51:11.595799 20087 raft_consensus.cc:504] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cb6a6aaa4845088804eb9b81414e43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34665 } } peers { permanent_uuid: "2cd15a8b0ca745608ac5bccb1f3a3fd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44971 } } peers { permanent_uuid: "47861012027e456f9dfec8bdb6ca88fd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42791 } }
I20260927 00:51:11.597010 20087 leader_election.cc:288] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cd15a8b0ca745608ac5bccb1f3a3fd6 (127.0.98.3:44971), 47861012027e456f9dfec8bdb6ca88fd (127.0.98.1:42791)
I20260927 00:51:11.601372 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:11.597430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54927 (local address 127.0.98.3:44971)
0927 00:51:11.597577 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:11.597581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:11.597763 (+ 182us) server_negotiation.cc:407] Connection header received
0927 00:51:11.597942 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:11.597945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:11.597988 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:11.598072 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:11.598386 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.598909 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.600192 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.600749 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.600877 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:11.600975 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:11.601153 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 00:51:11.601223 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 00:51:11.601401 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:11.597470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34919 (local address 127.0.98.1:42791)
0927 00:51:11.597609 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:11.597616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:11.597823 (+ 207us) server_negotiation.cc:407] Connection header received
0927 00:51:11.597981 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:11.597989 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:11.598032 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:11.598117 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:11.598426 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.598930 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.600190 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.600826 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.600948 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:11.601013 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:11.601195 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 00:51:11.601271 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:51:11.601416 20088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:11.597278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42791 (local address 127.0.98.2:34919)
0927 00:51:11.597714 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:11.597726 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:11.597864 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:11.598139 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:11.598142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:11.598157 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:11.598318 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:11.598324 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.599039 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.599044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:11.600063 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:11.600074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.600952 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.600955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:11.601062 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:11.601090 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 00:51:11.601179 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 00:51:11.601259 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":138,"thread_start_us":82,"threads_started":1}
I20260927 00:51:11.601421 20089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:11.597278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44971 (local address 127.0.98.2:54927)
0927 00:51:11.597653 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:11.597666 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:11.597819 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:11.598094 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:11.598097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:11.598113 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:11.598269 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:11.598275 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.599029 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.599036 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:11.600072 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:11.600082 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.600885 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.600888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:11.600996 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:11.601031 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 00:51:11.601134 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 00:51:11.601213 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":43,"thread_start_us":85,"threads_started":1}
I20260927 00:51:11.602077 20016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23a68638c57e4002aa5cbc7d915dd3de" candidate_uuid: "47cb6a6aaa4845088804eb9b81414e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd15a8b0ca745608ac5bccb1f3a3fd6" is_pre_election: true
I20260927 00:51:11.602146 19748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23a68638c57e4002aa5cbc7d915dd3de" candidate_uuid: "47cb6a6aaa4845088804eb9b81414e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47861012027e456f9dfec8bdb6ca88fd" is_pre_election: true
I20260927 00:51:11.602424 20016 raft_consensus.cc:2454] T 23a68638c57e4002aa5cbc7d915dd3de P 2cd15a8b0ca745608ac5bccb1f3a3fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47cb6a6aaa4845088804eb9b81414e43 in term 0.
I20260927 00:51:11.602432 19748 raft_consensus.cc:2454] T 23a68638c57e4002aa5cbc7d915dd3de P 47861012027e456f9dfec8bdb6ca88fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47cb6a6aaa4845088804eb9b81414e43 in term 0.
I20260927 00:51:11.602914 19818 leader_election.cc:302] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cd15a8b0ca745608ac5bccb1f3a3fd6, 47cb6a6aaa4845088804eb9b81414e43; no voters:
I20260927 00:51:11.603226 20087 raft_consensus.cc:2781] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:51:11.603354 20087 raft_consensus.cc:482] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:51:11.603387 20087 raft_consensus.cc:3037] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:11.604455 20087 raft_consensus.cc:504] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cb6a6aaa4845088804eb9b81414e43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34665 } } peers { permanent_uuid: "2cd15a8b0ca745608ac5bccb1f3a3fd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44971 } } peers { permanent_uuid: "47861012027e456f9dfec8bdb6ca88fd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42791 } }
I20260927 00:51:11.604813 20087 leader_election.cc:288] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43 [CANDIDATE]: Term 1 election: Requested vote from peers 2cd15a8b0ca745608ac5bccb1f3a3fd6 (127.0.98.3:44971), 47861012027e456f9dfec8bdb6ca88fd (127.0.98.1:42791)
I20260927 00:51:11.605170 19748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23a68638c57e4002aa5cbc7d915dd3de" candidate_uuid: "47cb6a6aaa4845088804eb9b81414e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47861012027e456f9dfec8bdb6ca88fd"
I20260927 00:51:11.605286 19748 raft_consensus.cc:3037] T 23a68638c57e4002aa5cbc7d915dd3de P 47861012027e456f9dfec8bdb6ca88fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:11.606313 19748 raft_consensus.cc:2454] T 23a68638c57e4002aa5cbc7d915dd3de P 47861012027e456f9dfec8bdb6ca88fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47cb6a6aaa4845088804eb9b81414e43 in term 1.
I20260927 00:51:11.606642 19816 leader_election.cc:302] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47861012027e456f9dfec8bdb6ca88fd, 47cb6a6aaa4845088804eb9b81414e43; no voters:
I20260927 00:51:11.606614 20016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23a68638c57e4002aa5cbc7d915dd3de" candidate_uuid: "47cb6a6aaa4845088804eb9b81414e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cd15a8b0ca745608ac5bccb1f3a3fd6"
I20260927 00:51:11.606765 20016 raft_consensus.cc:3037] T 23a68638c57e4002aa5cbc7d915dd3de P 2cd15a8b0ca745608ac5bccb1f3a3fd6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:11.606829 20087 raft_consensus.cc:2781] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:11.607051 20087 raft_consensus.cc:686] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43 [term 1 LEADER]: Becoming Leader. State: Replica: 47cb6a6aaa4845088804eb9b81414e43, State: Running, Role: LEADER
I20260927 00:51:11.607448 20087 consensus_queue.cc:237] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cb6a6aaa4845088804eb9b81414e43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34665 } } peers { permanent_uuid: "2cd15a8b0ca745608ac5bccb1f3a3fd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44971 } } peers { permanent_uuid: "47861012027e456f9dfec8bdb6ca88fd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42791 } }
I20260927 00:51:11.608132 20016 raft_consensus.cc:2454] T 23a68638c57e4002aa5cbc7d915dd3de P 2cd15a8b0ca745608ac5bccb1f3a3fd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47cb6a6aaa4845088804eb9b81414e43 in term 1.
I20260927 00:51:11.611132 19603 catalog_manager.cc:5495] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47cb6a6aaa4845088804eb9b81414e43 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "47cb6a6aaa4845088804eb9b81414e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cb6a6aaa4845088804eb9b81414e43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cd15a8b0ca745608ac5bccb1f3a3fd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47861012027e456f9dfec8bdb6ca88fd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42791 } health_report { overall_health: UNKNOWN } } }
I20260927 00:51:11.626570 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:11.622811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43656 (local address 127.0.98.2:34665)
0927 00:51:11.622946 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:11.622951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:11.622965 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:51:11.623104 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:11.623106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:11.623164 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:11.623316 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:51:11.623564 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.624077 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.625050 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.625281 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.625394 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:11.625721 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:51:11.626167 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:51:11.626315 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:11.626349 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 00:51:11.626411 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 00:51:11.632867 19748 raft_consensus.cc:1261] T 23a68638c57e4002aa5cbc7d915dd3de P 47861012027e456f9dfec8bdb6ca88fd [term 1 FOLLOWER]: Refusing update from remote peer 47cb6a6aaa4845088804eb9b81414e43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:51:11.632867 20016 raft_consensus.cc:1261] T 23a68638c57e4002aa5cbc7d915dd3de P 2cd15a8b0ca745608ac5bccb1f3a3fd6 [term 1 FOLLOWER]: Refusing update from remote peer 47cb6a6aaa4845088804eb9b81414e43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:51:11.633627 20090 consensus_queue.cc:1035] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47861012027e456f9dfec8bdb6ca88fd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:51:11.633845 20087 consensus_queue.cc:1035] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cd15a8b0ca745608ac5bccb1f3a3fd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:51:11.642740 20094 mvcc.cc:205] Tried to move back new op lower bound from 7333766232600399872 to 7333766232507777024. Current Snapshot: MvccSnapshot[applied={T|T < 7333766232600399872}]
I20260927 00:51:11.651741 19603 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:51:11.651953 19603 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:51:11.654774 19603 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fdd4f20e57c4e2db17d8bbffd04e5c2: Sending DeleteTablet for 3 replicas of tablet 23a68638c57e4002aa5cbc7d915dd3de
I20260927 00:51:11.655743 19862 tablet_service.cc:1514] Processing DeleteTablet for tablet 23a68638c57e4002aa5cbc7d915dd3de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:51:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43654
I20260927 00:51:11.655764 19728 tablet_service.cc:1514] Processing DeleteTablet for tablet 23a68638c57e4002aa5cbc7d915dd3de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:51:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57016
I20260927 00:51:11.655917 19996 tablet_service.cc:1514] Processing DeleteTablet for tablet 23a68638c57e4002aa5cbc7d915dd3de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:51:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33930
I20260927 00:51:11.656402 20105 tablet_replica.cc:330] T 23a68638c57e4002aa5cbc7d915dd3de P 2cd15a8b0ca745608ac5bccb1f3a3fd6: stopping tablet replica
I20260927 00:51:11.656859 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 19665
I20260927 00:51:11.659107 20105 raft_consensus.cc:2229] T 23a68638c57e4002aa5cbc7d915dd3de P 2cd15a8b0ca745608ac5bccb1f3a3fd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:51:11.659551 20105 raft_consensus.cc:2258] T 23a68638c57e4002aa5cbc7d915dd3de P 2cd15a8b0ca745608ac5bccb1f3a3fd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:51:11.660492 20104 tablet_replica.cc:330] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43: stopping tablet replica
I20260927 00:51:11.660822 20104 raft_consensus.cc:2229] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:51:11.661226 20104 raft_consensus.cc:2258] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:51:11.661252 20105 ts_tablet_manager.cc:1905] T 23a68638c57e4002aa5cbc7d915dd3de P 2cd15a8b0ca745608ac5bccb1f3a3fd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:51:11.664178 20105 ts_tablet_manager.cc:1918] T 23a68638c57e4002aa5cbc7d915dd3de P 2cd15a8b0ca745608ac5bccb1f3a3fd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:51:11.664314 20105 log.cc:1196] T 23a68638c57e4002aa5cbc7d915dd3de P 2cd15a8b0ca745608ac5bccb1f3a3fd6: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790470006085796-392-0/minicluster-data/ts-2/wal/wals/23a68638c57e4002aa5cbc7d915dd3de
I20260927 00:51:11.664680 20105 ts_tablet_manager.cc:1939] T 23a68638c57e4002aa5cbc7d915dd3de P 2cd15a8b0ca745608ac5bccb1f3a3fd6: Deleting consensus metadata
I20260927 00:51:11.666217 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 19799
W20260927 00:51:11.666205 19588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42791 (error 108)
I20260927 00:51:11.666231 20104 ts_tablet_manager.cc:1905] T 23a68638c57e4002aa5cbc7d915dd3de P 47cb6a6aaa4845088804eb9b81414e43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:51:11.667233 19590 catalog_manager.cc:4841] TS 2cd15a8b0ca745608ac5bccb1f3a3fd6 (127.0.98.3:44971): tablet 23a68638c57e4002aa5cbc7d915dd3de (table test-table [id=3daea2e536a044488c34591a8dbc98ad]) successfully deleted
I20260927 00:51:11.668016 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:11.667629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42791 (local address 127.0.0.1:57018)
0927 00:51:11.667828 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:11.667907 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260927 00:51:11.668164 19588 catalog_manager.cc:4584] TS 47861012027e456f9dfec8bdb6ca88fd (127.0.98.1:42791): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 23a68638c57e4002aa5cbc7d915dd3de: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42791: connect: Connection refused (error 111)
I20260927 00:51:11.672842 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 19933
I20260927 00:51:11.673689 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:11.672974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34665 (local address 127.0.0.1:43670)
0927 00:51:11.673485 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:11.673556 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396}
I20260927 00:51:11.678761 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 19571
2026-09-27T00:51:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2534 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T00:51:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:51:11Z Disabled control of system clock
I20260927 00:51:11.710345 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44067
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:51:11.826570 20110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:11.826879 20110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:11.826918 20110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:11.831048 20110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:11.831125 20110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:11.831139 20110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:11.831157 20110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:11.831172 20110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:11.836277 20110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44067
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:11.837409 20110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:11.838850 20110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:11.847857 20115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:11.848049 20118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:11.848069 20116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:11.848263 20110 server_base.cc:1034] running on GCE node
I20260927 00:51:11.848835 20110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:11.850049 20110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:11.851285 20110 hybrid_clock.cc:648] HybridClock initialized: now 1790470271851233 us; error 49 us; skew 500 ppm
I20260927 00:51:11.853533 20110 webserver.cc:460] Webserver started at http://127.0.98.62:40267/ using document root <none> and password file <none>
I20260927 00:51:11.854187 20110 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:11.854298 20110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:11.854548 20110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:11.856931 20110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "005cb63d21f045e08458c15d450f6aa8"
format_stamp: "Formatted at 2026-09-27 00:51:11 on dist-test-slave-86rs"
server_key: "cf79928f9b8ece7c6fd7bec1f85c1d0c"
server_key_iv: "58c4ee9558b7be7a0fbe508077f2017e"
server_key_version: "encryptionkey@0"
I20260927 00:51:11.857553 20110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "005cb63d21f045e08458c15d450f6aa8"
format_stamp: "Formatted at 2026-09-27 00:51:11 on dist-test-slave-86rs"
server_key: "cf79928f9b8ece7c6fd7bec1f85c1d0c"
server_key_iv: "58c4ee9558b7be7a0fbe508077f2017e"
server_key_version: "encryptionkey@0"
I20260927 00:51:11.861567 20110 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:51:11.864284 20124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:11.865701 20110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:51:11.865857 20110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "005cb63d21f045e08458c15d450f6aa8"
format_stamp: "Formatted at 2026-09-27 00:51:11 on dist-test-slave-86rs"
server_key: "cf79928f9b8ece7c6fd7bec1f85c1d0c"
server_key_iv: "58c4ee9558b7be7a0fbe508077f2017e"
server_key_version: "encryptionkey@0"
I20260927 00:51:11.865991 20110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:11.879376 20110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:11.880146 20110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:11.880395 20110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:11.887575 20110 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44067
I20260927 00:51:11.887586 20176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44067 every 8 connection(s)
I20260927 00:51:11.888645 20110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:51:11.891848 20177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:11.896759 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 20110
I20260927 00:51:11.896875 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:51:11.897219 392 external_mini_cluster.cc:1428] Setting key e553b8a5b1a4e45645fd94ebd2763726
I20260927 00:51:11.898078 20177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 005cb63d21f045e08458c15d450f6aa8: Bootstrap starting.
I20260927 00:51:11.900671 20177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 005cb63d21f045e08458c15d450f6aa8: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:11.901532 20177 log.cc:824] T 00000000000000000000000000000000 P 005cb63d21f045e08458c15d450f6aa8: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:11.903590 20177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 005cb63d21f045e08458c15d450f6aa8: No bootstrap required, opened a new log
I20260927 00:51:11.907253 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:11.898780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49752 (local address 127.0.98.62:44067)
0927 00:51:11.899490 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:11.899505 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:11.899543 (+ 38us) server_negotiation.cc:407] Connection header received
0927 00:51:11.900308 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:11.900335 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:11.900825 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:11.901192 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:11.901745 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.902725 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.903701 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:11.904199 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:11.904396 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:11.905313 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:11.905319 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:11.905333 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:11.905747 (+ 414us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:11.905807 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:11.905829 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:11.905906 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:11.906074 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 00:51:11.906380 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":147,"threads_started":1}
I20260927 00:51:11.908041 20177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 005cb63d21f045e08458c15d450f6aa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "005cb63d21f045e08458c15d450f6aa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44067 } }
I20260927 00:51:11.908365 20177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 005cb63d21f045e08458c15d450f6aa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:11.908429 20177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 005cb63d21f045e08458c15d450f6aa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 005cb63d21f045e08458c15d450f6aa8, State: Initialized, Role: FOLLOWER
I20260927 00:51:11.908910 20177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 005cb63d21f045e08458c15d450f6aa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "005cb63d21f045e08458c15d450f6aa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44067 } }
I20260927 00:51:11.909076 20177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 005cb63d21f045e08458c15d450f6aa8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:51:11.909132 20177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 005cb63d21f045e08458c15d450f6aa8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:51:11.909224 20177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 005cb63d21f045e08458c15d450f6aa8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:11.910260 20177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 005cb63d21f045e08458c15d450f6aa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "005cb63d21f045e08458c15d450f6aa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44067 } }
I20260927 00:51:11.910567 20177 leader_election.cc:302] T 00000000000000000000000000000000 P 005cb63d21f045e08458c15d450f6aa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 005cb63d21f045e08458c15d450f6aa8; no voters:
I20260927 00:51:11.910835 20177 leader_election.cc:288] T 00000000000000000000000000000000 P 005cb63d21f045e08458c15d450f6aa8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:51:11.911067 20182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 005cb63d21f045e08458c15d450f6aa8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:11.911391 20182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 005cb63d21f045e08458c15d450f6aa8 [term 1 LEADER]: Becoming Leader. State: Replica: 005cb63d21f045e08458c15d450f6aa8, State: Running, Role: LEADER
I20260927 00:51:11.911753 20182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 005cb63d21f045e08458c15d450f6aa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "005cb63d21f045e08458c15d450f6aa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44067 } }
I20260927 00:51:11.912119 20177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 005cb63d21f045e08458c15d450f6aa8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:51:11.913460 20184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 005cb63d21f045e08458c15d450f6aa8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 005cb63d21f045e08458c15d450f6aa8. Latest consensus state: current_term: 1 leader_uuid: "005cb63d21f045e08458c15d450f6aa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "005cb63d21f045e08458c15d450f6aa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44067 } } }
I20260927 00:51:11.913592 20184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 005cb63d21f045e08458c15d450f6aa8 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:11.913929 20183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 005cb63d21f045e08458c15d450f6aa8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "005cb63d21f045e08458c15d450f6aa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "005cb63d21f045e08458c15d450f6aa8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44067 } } }
I20260927 00:51:11.914047 20192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:51:11.914081 20183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 005cb63d21f045e08458c15d450f6aa8 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:11.917388 20192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:51:11.923188 20192 catalog_manager.cc:1347] Generated new cluster ID: 7cc4bb15d35f444ab447a41fe469cffa
I20260927 00:51:11.923283 20192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:51:11.943127 20192 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:51:11.944586 20192 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:51:11.951879 20192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 005cb63d21f045e08458c15d450f6aa8: Generated new TSK 0
I20260927 00:51:11.952883 20192 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:51:11.957477 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44067
--builtin_ntp_servers=127.0.98.20:44209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:51:12.078533 20201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:12.078866 20201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:12.078943 20201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:12.083326 20201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:12.083438 20201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:12.083534 20201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:51:12.088761 20201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:12.090013 20201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:12.091526 20201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:12.102366 20206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:12.102366 20207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:12.102366 20209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:12.102838 20201 server_base.cc:1034] running on GCE node
I20260927 00:51:12.103292 20201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:12.104009 20201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:12.105206 20201 hybrid_clock.cc:648] HybridClock initialized: now 1790470272105152 us; error 65 us; skew 500 ppm
I20260927 00:51:12.107291 20201 webserver.cc:460] Webserver started at http://127.0.98.1:41705/ using document root <none> and password file <none>
I20260927 00:51:12.107878 20201 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:12.107944 20201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:12.108124 20201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:12.110400 20201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-0/data/instance:
uuid: "6fa9d14101b54852b3a1c09a86a264a8"
format_stamp: "Formatted at 2026-09-27 00:51:12 on dist-test-slave-86rs"
server_key: "f32373bab75bcb8adc7d1c883e0e0071"
server_key_iv: "27cfcee98d65c87b9234dec2abbe07de"
server_key_version: "encryptionkey@0"
I20260927 00:51:12.110975 20201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6fa9d14101b54852b3a1c09a86a264a8"
format_stamp: "Formatted at 2026-09-27 00:51:12 on dist-test-slave-86rs"
server_key: "f32373bab75bcb8adc7d1c883e0e0071"
server_key_iv: "27cfcee98d65c87b9234dec2abbe07de"
server_key_version: "encryptionkey@0"
I20260927 00:51:12.115419 20201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:51:12.118470 20215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:12.119915 20201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 00:51:12.120085 20201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-0/wal
uuid: "6fa9d14101b54852b3a1c09a86a264a8"
format_stamp: "Formatted at 2026-09-27 00:51:12 on dist-test-slave-86rs"
server_key: "f32373bab75bcb8adc7d1c883e0e0071"
server_key_iv: "27cfcee98d65c87b9234dec2abbe07de"
server_key_version: "encryptionkey@0"
I20260927 00:51:12.120226 20201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:12.155095 20201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:12.155920 20201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:12.156155 20201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:12.156886 20201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:51:12.157981 20201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:51:12.158058 20201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:12.158134 20201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:51:12.158186 20201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:12.169001 20201 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33831
I20260927 00:51:12.169023 20328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33831 every 8 connection(s)
I20260927 00:51:12.170083 20201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:51:12.175511 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 20201
I20260927 00:51:12.175623 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:51:12.175935 392 external_mini_cluster.cc:1428] Setting key d90959909d71e1a0f65736a214242a5b
I20260927 00:51:12.178177 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44067
--builtin_ntp_servers=127.0.98.20:44209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:51:12.180630 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:12.172196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59783 (local address 127.0.98.62:44067)
0927 00:51:12.172426 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:12.172430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:12.173535 (+ 1105us) server_negotiation.cc:407] Connection header received
0927 00:51:12.175004 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:12.175008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:12.175065 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:12.175148 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:12.176275 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.177035 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.178402 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.178638 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.178753 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:12.180020 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:12.180024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:12.180029 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:12.180048 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:12.180069 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:12.180078 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:12.180158 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:12.180403 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 00:51:12.180476 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260927 00:51:12.181613 20331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:12.172644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44067 (local address 127.0.98.1:59783)
0927 00:51:12.173339 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:12.173407 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:12.174757 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:12.175328 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:12.175335 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:12.175455 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:12.176085 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:12.176100 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.177169 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.177172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:12.178184 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:12.178194 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.178813 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.178817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:12.179016 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:12.179716 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:51:12.179718 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:12.179764 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:51:12.179838 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:12.180172 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:12.180182 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:12.180189 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:51:12.180403 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 00:51:12.180733 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":119,"threads_started":1}
I20260927 00:51:12.183054 20329 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44067
I20260927 00:51:12.183391 20329 heartbeater.cc:464] Registering TS with master...
I20260927 00:51:12.184196 20329 heartbeater.cc:511] Master 127.0.98.62:44067 requested a full tablet report, sending...
I20260927 00:51:12.186118 20141 ts_manager.cc:194] Registered new tserver with Master: 6fa9d14101b54852b3a1c09a86a264a8 (127.0.98.1:33831)
I20260927 00:51:12.187381 20141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:59783
W20260927 00:51:12.295588 20332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:12.295888 20332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:12.295929 20332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:12.300211 20332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:12.300308 20332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:12.300367 20332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:51:12.305433 20332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:12.306668 20332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:12.308156 20332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:12.318795 20338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:12.318825 20337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:12.319077 20332 server_base.cc:1034] running on GCE node
W20260927 00:51:12.318799 20340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:12.319636 20332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:12.320436 20332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:12.321655 20332 hybrid_clock.cc:648] HybridClock initialized: now 1790470272321633 us; error 34 us; skew 500 ppm
I20260927 00:51:12.323987 20332 webserver.cc:460] Webserver started at http://127.0.98.2:45157/ using document root <none> and password file <none>
I20260927 00:51:12.324820 20332 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:12.324913 20332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:12.325160 20332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:12.327586 20332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-1/data/instance:
uuid: "d5a3ca67162840399871c5a11b5bfbc7"
format_stamp: "Formatted at 2026-09-27 00:51:12 on dist-test-slave-86rs"
server_key: "9aee0d1fff1d63a2efd0bcd1e2834a65"
server_key_iv: "9181fa1357bb5b553da29a1d45f8b06c"
server_key_version: "encryptionkey@0"
I20260927 00:51:12.328194 20332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d5a3ca67162840399871c5a11b5bfbc7"
format_stamp: "Formatted at 2026-09-27 00:51:12 on dist-test-slave-86rs"
server_key: "9aee0d1fff1d63a2efd0bcd1e2834a65"
server_key_iv: "9181fa1357bb5b553da29a1d45f8b06c"
server_key_version: "encryptionkey@0"
I20260927 00:51:12.332365 20332 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:51:12.335095 20346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:12.336565 20332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:51:12.336727 20332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-1/wal
uuid: "d5a3ca67162840399871c5a11b5bfbc7"
format_stamp: "Formatted at 2026-09-27 00:51:12 on dist-test-slave-86rs"
server_key: "9aee0d1fff1d63a2efd0bcd1e2834a65"
server_key_iv: "9181fa1357bb5b553da29a1d45f8b06c"
server_key_version: "encryptionkey@0"
I20260927 00:51:12.336866 20332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:12.359501 20332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:12.360399 20332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:12.360656 20332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:12.361320 20332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:51:12.362334 20332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:51:12.362382 20332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:12.362455 20332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:51:12.362488 20332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:12.371471 20332 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42981
I20260927 00:51:12.371510 20459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42981 every 8 connection(s)
I20260927 00:51:12.372503 20332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:51:12.375823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 20332
I20260927 00:51:12.375936 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:51:12.376318 392 external_mini_cluster.cc:1428] Setting key b0c42735d5374988c5fa96fbc8a9604f
I20260927 00:51:12.378794 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44067
--builtin_ntp_servers=127.0.98.20:44209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:51:12.384644 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:12.374407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58221 (local address 127.0.98.62:44067)
0927 00:51:12.374584 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:12.374589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:12.375693 (+ 1104us) server_negotiation.cc:407] Connection header received
0927 00:51:12.377481 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:12.377485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:12.377539 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:12.377647 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:12.378884 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.379573 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.381723 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.381965 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.382065 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:12.383799 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:12.383802 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:12.383805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:12.383820 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:12.383844 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:12.383853 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:12.383928 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:12.384298 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 00:51:12.384384 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 00:51:12.386237 20462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:12.374831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44067 (local address 127.0.98.2:58221)
0927 00:51:12.375507 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:12.375572 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:12.377245 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:12.377848 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:12.377861 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:12.377979 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:12.378666 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:12.378683 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.379729 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.379733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:12.381539 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:12.381558 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.382167 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.382175 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:12.382455 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:12.383342 (+ 887us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:51:12.383347 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:12.383425 (+ 78us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:51:12.383553 (+ 128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:12.383990 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:12.384005 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:12.384016 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 00:51:12.384210 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 00:51:12.384882 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":122,"threads_started":1}
I20260927 00:51:12.387708 20460 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44067
I20260927 00:51:12.388000 20460 heartbeater.cc:464] Registering TS with master...
I20260927 00:51:12.388664 20460 heartbeater.cc:511] Master 127.0.98.62:44067 requested a full tablet report, sending...
I20260927 00:51:12.389865 20141 ts_manager.cc:194] Registered new tserver with Master: d5a3ca67162840399871c5a11b5bfbc7 (127.0.98.2:42981)
I20260927 00:51:12.390523 20141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:58221
W20260927 00:51:12.504670 20463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:12.505007 20463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:12.505075 20463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:12.509419 20463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:12.509516 20463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:12.509610 20463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:51:12.514982 20463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:12.516328 20463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:12.518015 20463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:12.528421 20468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:12.528585 20463 server_base.cc:1034] running on GCE node
W20260927 00:51:12.528405 20469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:12.528407 20471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:12.529198 20463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:12.529896 20463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:12.531052 20463 hybrid_clock.cc:648] HybridClock initialized: now 1790470272531023 us; error 43 us; skew 500 ppm
I20260927 00:51:12.533090 20463 webserver.cc:460] Webserver started at http://127.0.98.3:33069/ using document root <none> and password file <none>
I20260927 00:51:12.533699 20463 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:12.533762 20463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:12.533993 20463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:12.536280 20463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-2/data/instance:
uuid: "e80e66aad2284f098ed8b25cdefa1ff9"
format_stamp: "Formatted at 2026-09-27 00:51:12 on dist-test-slave-86rs"
server_key: "64f6ddbafecc5b78704b2849c9048a69"
server_key_iv: "cd83c58a8c10aa51224988be05ede8ac"
server_key_version: "encryptionkey@0"
I20260927 00:51:12.536870 20463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e80e66aad2284f098ed8b25cdefa1ff9"
format_stamp: "Formatted at 2026-09-27 00:51:12 on dist-test-slave-86rs"
server_key: "64f6ddbafecc5b78704b2849c9048a69"
server_key_iv: "cd83c58a8c10aa51224988be05ede8ac"
server_key_version: "encryptionkey@0"
I20260927 00:51:12.540705 20463 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:51:12.543304 20477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:12.544641 20463 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:51:12.544763 20463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-2/wal
uuid: "e80e66aad2284f098ed8b25cdefa1ff9"
format_stamp: "Formatted at 2026-09-27 00:51:12 on dist-test-slave-86rs"
server_key: "64f6ddbafecc5b78704b2849c9048a69"
server_key_iv: "cd83c58a8c10aa51224988be05ede8ac"
server_key_version: "encryptionkey@0"
I20260927 00:51:12.544879 20463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:12.560130 20463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:12.560979 20463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:12.561198 20463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:12.561842 20463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:51:12.562846 20463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:51:12.562893 20463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:12.562963 20463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:51:12.562996 20463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:12.572046 20463 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43259
I20260927 00:51:12.572108 20590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43259 every 8 connection(s)
I20260927 00:51:12.573040 20463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:51:12.576499 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 20463
I20260927 00:51:12.576591 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:51:12.576857 392 external_mini_cluster.cc:1428] Setting key 4edcf790d4e671525a610263e32ea043
I20260927 00:51:12.582695 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:12.574899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53783 (local address 127.0.98.62:44067)
0927 00:51:12.575057 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:12.575061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:12.576095 (+ 1034us) server_negotiation.cc:407] Connection header received
0927 00:51:12.577807 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:12.577811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:12.577875 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:12.577966 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:12.579099 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.579640 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.580794 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.581019 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.581105 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:12.582223 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:12.582227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:12.582229 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:12.582242 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:12.582262 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:12.582271 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:12.582341 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:12.582503 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 00:51:12.582573 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 00:51:12.583627 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:12.575306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44067 (local address 127.0.98.3:53783)
0927 00:51:12.575971 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:12.576012 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:12.577548 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:12.578189 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:12.578196 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:12.578327 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:12.578930 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:12.578960 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.579759 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.579761 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:12.580687 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:12.580696 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.581164 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.581167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:12.581338 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:12.581964 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:51:12.581965 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:12.582008 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:51:12.582076 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:12.582357 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:12.582363 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:12.582367 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:51:12.582473 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 00:51:12.582714 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":120,"threads_started":1}
I20260927 00:51:12.584914 20591 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44067
I20260927 00:51:12.585186 20591 heartbeater.cc:464] Registering TS with master...
I20260927 00:51:12.585850 20591 heartbeater.cc:511] Master 127.0.98.62:44067 requested a full tablet report, sending...
I20260927 00:51:12.587134 20141 ts_manager.cc:194] Registered new tserver with Master: e80e66aad2284f098ed8b25cdefa1ff9 (127.0.98.3:43259)
I20260927 00:51:12.587746 20141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:53783
I20260927 00:51:12.590798 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:51:12.596855 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:12.592794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49758 (local address 127.0.98.62:44067)
0927 00:51:12.592966 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:12.592972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:12.593152 (+ 180us) server_negotiation.cc:407] Connection header received
0927 00:51:12.593335 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:12.593339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:12.593393 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:12.593479 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:12.593768 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.594307 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.595182 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.595429 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.595541 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:12.596363 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:12.596367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:12.596370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:12.596383 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:12.596411 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:12.596419 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:12.596513 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:12.596639 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 00:51:12.596700 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 00:51:12.599829 20141 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:51:12.603001 20141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:51:12.617914 20602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:12.611711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43259 (local address 127.0.0.1:37452)
0927 00:51:12.612677 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:12.612690 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:12.612755 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:12.613588 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:12.613591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:12.613610 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:12.613709 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:12.613717 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.614844 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.614850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:12.616110 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:12.616120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.617313 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.617317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:12.617426 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:12.617471 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 00:51:12.617598 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 00:51:12.617731 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"mutex_wait_us":1,"thread_start_us":123,"threads_started":1}
I20260927 00:51:12.617941 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:12.611956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37452 (local address 127.0.98.3:43259)
0927 00:51:12.613088 (+ 1132us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:12.613119 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:12.613142 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:51:12.613214 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:12.613219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:12.613434 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:12.613599 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:12.613841 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.614732 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.616280 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.617134 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.617293 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:12.617508 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:12.617644 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 00:51:12.617762 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1032,"thread_start_us":144,"threads_started":1}
I20260927 00:51:12.618402 20606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:12.612415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42981 (local address 127.0.0.1:46410)
0927 00:51:12.613401 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:12.613414 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:12.613533 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:12.614050 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:12.614053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:12.614073 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:12.614162 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:12.614168 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.615560 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.615564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:12.616690 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:12.616701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.617999 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.618003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:12.618103 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:12.618128 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:51:12.618194 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 00:51:12.618268 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"thread_start_us":84,"threads_started":1}
I20260927 00:51:12.618948 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:12.611687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33831 (local address 127.0.0.1:55474)
0927 00:51:12.612527 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:12.612580 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:12.612720 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:12.613356 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:12.613363 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:12.613388 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:12.613584 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:12.613591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.615128 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.615131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:12.616690 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:12.616701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.618389 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.618392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:12.618658 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:12.618683 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:51:12.618747 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 00:51:12.618811 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":25,"thread_start_us":81,"threads_started":1}
I20260927 00:51:12.618988 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:12.612808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46410 (local address 127.0.98.2:42981)
0927 00:51:12.613229 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:12.613238 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:12.613633 (+ 395us) server_negotiation.cc:407] Connection header received
0927 00:51:12.613716 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:12.613721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:12.613904 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:12.614059 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:12.614288 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.615424 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.616817 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.617644 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.617747 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:12.618671 (+ 924us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:12.618721 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 00:51:12.618807 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":110,"threads_started":1}
I20260927 00:51:12.619529 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:12.612134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55474 (local address 127.0.98.1:33831)
0927 00:51:12.612537 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:12.612549 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:12.612869 (+ 320us) server_negotiation.cc:407] Connection header received
0927 00:51:12.612962 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:12.612968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:12.613190 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:12.613347 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:12.613716 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.614994 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.616850 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.617732 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.617833 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:12.619241 (+ 1408us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:12.619306 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 00:51:12.619408 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":119,"threads_started":1}
I20260927 00:51:12.621359 20525 tablet_service.cc:1467] Processing CreateTablet for tablet dcce3f5de7954105bb73631074f16ef9 (DEFAULT_TABLE table=test-table [id=76e1cd13387c4fc5a78d817986e150d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:51:12.621840 20263 tablet_service.cc:1467] Processing CreateTablet for tablet dcce3f5de7954105bb73631074f16ef9 (DEFAULT_TABLE table=test-table [id=76e1cd13387c4fc5a78d817986e150d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:51:12.621831 20394 tablet_service.cc:1467] Processing CreateTablet for tablet dcce3f5de7954105bb73631074f16ef9 (DEFAULT_TABLE table=test-table [id=76e1cd13387c4fc5a78d817986e150d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:51:12.622567 20525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcce3f5de7954105bb73631074f16ef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:12.622798 20394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcce3f5de7954105bb73631074f16ef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:12.622852 20263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcce3f5de7954105bb73631074f16ef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:12.629282 20608 tablet_bootstrap.cc:492] T dcce3f5de7954105bb73631074f16ef9 P 6fa9d14101b54852b3a1c09a86a264a8: Bootstrap starting.
I20260927 00:51:12.630738 20609 tablet_bootstrap.cc:492] T dcce3f5de7954105bb73631074f16ef9 P d5a3ca67162840399871c5a11b5bfbc7: Bootstrap starting.
I20260927 00:51:12.631877 20608 tablet_bootstrap.cc:654] T dcce3f5de7954105bb73631074f16ef9 P 6fa9d14101b54852b3a1c09a86a264a8: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:12.632624 20610 tablet_bootstrap.cc:492] T dcce3f5de7954105bb73631074f16ef9 P e80e66aad2284f098ed8b25cdefa1ff9: Bootstrap starting.
I20260927 00:51:12.632880 20608 log.cc:824] T dcce3f5de7954105bb73631074f16ef9 P 6fa9d14101b54852b3a1c09a86a264a8: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:12.633009 20609 tablet_bootstrap.cc:654] T dcce3f5de7954105bb73631074f16ef9 P d5a3ca67162840399871c5a11b5bfbc7: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:12.633834 20609 log.cc:824] T dcce3f5de7954105bb73631074f16ef9 P d5a3ca67162840399871c5a11b5bfbc7: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:12.634784 20610 tablet_bootstrap.cc:654] T dcce3f5de7954105bb73631074f16ef9 P e80e66aad2284f098ed8b25cdefa1ff9: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:12.635455 20610 log.cc:824] T dcce3f5de7954105bb73631074f16ef9 P e80e66aad2284f098ed8b25cdefa1ff9: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:12.635771 20608 tablet_bootstrap.cc:492] T dcce3f5de7954105bb73631074f16ef9 P 6fa9d14101b54852b3a1c09a86a264a8: No bootstrap required, opened a new log
I20260927 00:51:12.635771 20609 tablet_bootstrap.cc:492] T dcce3f5de7954105bb73631074f16ef9 P d5a3ca67162840399871c5a11b5bfbc7: No bootstrap required, opened a new log
I20260927 00:51:12.635987 20609 ts_tablet_manager.cc:1397] T dcce3f5de7954105bb73631074f16ef9 P d5a3ca67162840399871c5a11b5bfbc7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:51:12.635987 20608 ts_tablet_manager.cc:1397] T dcce3f5de7954105bb73631074f16ef9 P 6fa9d14101b54852b3a1c09a86a264a8: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 00:51:12.637360 20610 tablet_bootstrap.cc:492] T dcce3f5de7954105bb73631074f16ef9 P e80e66aad2284f098ed8b25cdefa1ff9: No bootstrap required, opened a new log
I20260927 00:51:12.637580 20610 ts_tablet_manager.cc:1397] T dcce3f5de7954105bb73631074f16ef9 P e80e66aad2284f098ed8b25cdefa1ff9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 00:51:12.640579 20609 raft_consensus.cc:348] T dcce3f5de7954105bb73631074f16ef9 P d5a3ca67162840399871c5a11b5bfbc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa9d14101b54852b3a1c09a86a264a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33831 } } peers { permanent_uuid: "e80e66aad2284f098ed8b25cdefa1ff9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43259 } } peers { permanent_uuid: "d5a3ca67162840399871c5a11b5bfbc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42981 } }
I20260927 00:51:12.640579 20608 raft_consensus.cc:348] T dcce3f5de7954105bb73631074f16ef9 P 6fa9d14101b54852b3a1c09a86a264a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa9d14101b54852b3a1c09a86a264a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33831 } } peers { permanent_uuid: "e80e66aad2284f098ed8b25cdefa1ff9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43259 } } peers { permanent_uuid: "d5a3ca67162840399871c5a11b5bfbc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42981 } }
I20260927 00:51:12.640882 20608 raft_consensus.cc:374] T dcce3f5de7954105bb73631074f16ef9 P 6fa9d14101b54852b3a1c09a86a264a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:12.640882 20609 raft_consensus.cc:374] T dcce3f5de7954105bb73631074f16ef9 P d5a3ca67162840399871c5a11b5bfbc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:12.640934 20608 raft_consensus.cc:729] T dcce3f5de7954105bb73631074f16ef9 P 6fa9d14101b54852b3a1c09a86a264a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa9d14101b54852b3a1c09a86a264a8, State: Initialized, Role: FOLLOWER
I20260927 00:51:12.640945 20609 raft_consensus.cc:729] T dcce3f5de7954105bb73631074f16ef9 P d5a3ca67162840399871c5a11b5bfbc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5a3ca67162840399871c5a11b5bfbc7, State: Initialized, Role: FOLLOWER
I20260927 00:51:12.640880 20610 raft_consensus.cc:348] T dcce3f5de7954105bb73631074f16ef9 P e80e66aad2284f098ed8b25cdefa1ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa9d14101b54852b3a1c09a86a264a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33831 } } peers { permanent_uuid: "e80e66aad2284f098ed8b25cdefa1ff9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43259 } } peers { permanent_uuid: "d5a3ca67162840399871c5a11b5bfbc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42981 } }
I20260927 00:51:12.641124 20610 raft_consensus.cc:374] T dcce3f5de7954105bb73631074f16ef9 P e80e66aad2284f098ed8b25cdefa1ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:12.641184 20610 raft_consensus.cc:729] T dcce3f5de7954105bb73631074f16ef9 P e80e66aad2284f098ed8b25cdefa1ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e80e66aad2284f098ed8b25cdefa1ff9, State: Initialized, Role: FOLLOWER
I20260927 00:51:12.641472 20609 consensus_queue.cc:260] T dcce3f5de7954105bb73631074f16ef9 P d5a3ca67162840399871c5a11b5bfbc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa9d14101b54852b3a1c09a86a264a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33831 } } peers { permanent_uuid: "e80e66aad2284f098ed8b25cdefa1ff9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43259 } } peers { permanent_uuid: "d5a3ca67162840399871c5a11b5bfbc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42981 } }
I20260927 00:51:12.641472 20608 consensus_queue.cc:260] T dcce3f5de7954105bb73631074f16ef9 P 6fa9d14101b54852b3a1c09a86a264a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa9d14101b54852b3a1c09a86a264a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33831 } } peers { permanent_uuid: "e80e66aad2284f098ed8b25cdefa1ff9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43259 } } peers { permanent_uuid: "d5a3ca67162840399871c5a11b5bfbc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42981 } }
I20260927 00:51:12.641748 20610 consensus_queue.cc:260] T dcce3f5de7954105bb73631074f16ef9 P e80e66aad2284f098ed8b25cdefa1ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa9d14101b54852b3a1c09a86a264a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33831 } } peers { permanent_uuid: "e80e66aad2284f098ed8b25cdefa1ff9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43259 } } peers { permanent_uuid: "d5a3ca67162840399871c5a11b5bfbc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42981 } }
I20260927 00:51:12.642362 20460 heartbeater.cc:503] Master 127.0.98.62:44067 was elected leader, sending a full tablet report...
I20260927 00:51:12.642582 20608 ts_tablet_manager.cc:1428] T dcce3f5de7954105bb73631074f16ef9 P 6fa9d14101b54852b3a1c09a86a264a8: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260927 00:51:12.642714 20610 ts_tablet_manager.cc:1428] T dcce3f5de7954105bb73631074f16ef9 P e80e66aad2284f098ed8b25cdefa1ff9: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:51:12.642714 20329 heartbeater.cc:503] Master 127.0.98.62:44067 was elected leader, sending a full tablet report...
I20260927 00:51:12.643056 20591 heartbeater.cc:503] Master 127.0.98.62:44067 was elected leader, sending a full tablet report...
I20260927 00:51:12.644083 20609 ts_tablet_manager.cc:1428] T dcce3f5de7954105bb73631074f16ef9 P d5a3ca67162840399871c5a11b5bfbc7: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
W20260927 00:51:12.671422 20330 tablet.cc:2369] T dcce3f5de7954105bb73631074f16ef9 P 6fa9d14101b54852b3a1c09a86a264a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:51:12.690727 20614 raft_consensus.cc:482] T dcce3f5de7954105bb73631074f16ef9 P 6fa9d14101b54852b3a1c09a86a264a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:51:12.690984 20614 raft_consensus.cc:504] T dcce3f5de7954105bb73631074f16ef9 P 6fa9d14101b54852b3a1c09a86a264a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa9d14101b54852b3a1c09a86a264a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33831 } } peers { permanent_uuid: "e80e66aad2284f098ed8b25cdefa1ff9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43259 } } peers { permanent_uuid: "d5a3ca67162840399871c5a11b5bfbc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42981 } }
I20260927 00:51:12.692178 20614 leader_election.cc:288] T dcce3f5de7954105bb73631074f16ef9 P 6fa9d14101b54852b3a1c09a86a264a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e80e66aad2284f098ed8b25cdefa1ff9 (127.0.98.3:43259), d5a3ca67162840399871c5a11b5bfbc7 (127.0.98.2:42981)
I20260927 00:51:12.696506 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:12.692688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55303 (local address 127.0.98.3:43259)
0927 00:51:12.692881 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:12.692887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:12.692916 (+ 29us) server_negotiation.cc:407] Connection header received
0927 00:51:12.693090 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:12.693093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:12.693150 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:12.693227 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:12.693598 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.694256 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.695446 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.695881 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.695969 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:12.696050 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:12.696281 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 00:51:12.696364 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 00:51:12.696537 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:12.693115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44257 (local address 127.0.98.2:42981)
0927 00:51:12.693259 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:12.693263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:12.693413 (+ 150us) server_negotiation.cc:407] Connection header received
0927 00:51:12.693501 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:12.693505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:12.693566 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:12.693641 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:51:12.693940 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.694477 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.695540 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.695957 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.696062 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:12.696121 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:12.696339 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 00:51:12.696406 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:51:12.696578 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:12.692948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42981 (local address 127.0.98.1:44257)
0927 00:51:12.693315 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:12.693329 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:12.693419 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:12.693672 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:12.693675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:12.693692 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:12.693823 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:12.693829 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.694584 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.694586 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:12.695402 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:12.695415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.696103 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.696106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:12.696175 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:12.696214 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 00:51:12.696322 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 00:51:12.696389 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":123,"threads_started":1}
I20260927 00:51:12.696587 20617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:12.692561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43259 (local address 127.0.98.1:55303)
0927 00:51:12.692832 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:12.692846 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:12.692976 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:12.693277 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:12.693281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:12.693307 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:51:12.693476 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:12.693484 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.694367 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.694370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:12.695316 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:12.695326 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.696011 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.696018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:12.696117 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:12.696150 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 00:51:12.696275 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 00:51:12.696358 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":24,"thread_start_us":88,"threads_started":1}
I20260927 00:51:12.697337 20414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcce3f5de7954105bb73631074f16ef9" candidate_uuid: "6fa9d14101b54852b3a1c09a86a264a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5a3ca67162840399871c5a11b5bfbc7" is_pre_election: true
I20260927 00:51:12.697338 20545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcce3f5de7954105bb73631074f16ef9" candidate_uuid: "6fa9d14101b54852b3a1c09a86a264a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e80e66aad2284f098ed8b25cdefa1ff9" is_pre_election: true
I20260927 00:51:12.697646 20414 raft_consensus.cc:2454] T dcce3f5de7954105bb73631074f16ef9 P d5a3ca67162840399871c5a11b5bfbc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fa9d14101b54852b3a1c09a86a264a8 in term 0.
I20260927 00:51:12.697649 20545 raft_consensus.cc:2454] T dcce3f5de7954105bb73631074f16ef9 P e80e66aad2284f098ed8b25cdefa1ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fa9d14101b54852b3a1c09a86a264a8 in term 0.
I20260927 00:51:12.698139 20217 leader_election.cc:302] T dcce3f5de7954105bb73631074f16ef9 P 6fa9d14101b54852b3a1c09a86a264a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fa9d14101b54852b3a1c09a86a264a8, d5a3ca67162840399871c5a11b5bfbc7; no voters:
I20260927 00:51:12.698439 20614 raft_consensus.cc:2781] T dcce3f5de7954105bb73631074f16ef9 P 6fa9d14101b54852b3a1c09a86a264a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:51:12.698536 20614 raft_consensus.cc:482] T dcce3f5de7954105bb73631074f16ef9 P 6fa9d14101b54852b3a1c09a86a264a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:51:12.698590 20614 raft_consensus.cc:3037] T dcce3f5de7954105bb73631074f16ef9 P 6fa9d14101b54852b3a1c09a86a264a8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:12.699838 20614 raft_consensus.cc:504] T dcce3f5de7954105bb73631074f16ef9 P 6fa9d14101b54852b3a1c09a86a264a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa9d14101b54852b3a1c09a86a264a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33831 } } peers { permanent_uuid: "e80e66aad2284f098ed8b25cdefa1ff9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43259 } } peers { permanent_uuid: "d5a3ca67162840399871c5a11b5bfbc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42981 } }
I20260927 00:51:12.700203 20614 leader_election.cc:288] T dcce3f5de7954105bb73631074f16ef9 P 6fa9d14101b54852b3a1c09a86a264a8 [CANDIDATE]: Term 1 election: Requested vote from peers e80e66aad2284f098ed8b25cdefa1ff9 (127.0.98.3:43259), d5a3ca67162840399871c5a11b5bfbc7 (127.0.98.2:42981)
I20260927 00:51:12.700598 20545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcce3f5de7954105bb73631074f16ef9" candidate_uuid: "6fa9d14101b54852b3a1c09a86a264a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e80e66aad2284f098ed8b25cdefa1ff9"
I20260927 00:51:12.700636 20414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcce3f5de7954105bb73631074f16ef9" candidate_uuid: "6fa9d14101b54852b3a1c09a86a264a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5a3ca67162840399871c5a11b5bfbc7"
I20260927 00:51:12.700722 20545 raft_consensus.cc:3037] T dcce3f5de7954105bb73631074f16ef9 P e80e66aad2284f098ed8b25cdefa1ff9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:12.700781 20414 raft_consensus.cc:3037] T dcce3f5de7954105bb73631074f16ef9 P d5a3ca67162840399871c5a11b5bfbc7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:12.701831 20545 raft_consensus.cc:2454] T dcce3f5de7954105bb73631074f16ef9 P e80e66aad2284f098ed8b25cdefa1ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa9d14101b54852b3a1c09a86a264a8 in term 1.
I20260927 00:51:12.701846 20414 raft_consensus.cc:2454] T dcce3f5de7954105bb73631074f16ef9 P d5a3ca67162840399871c5a11b5bfbc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fa9d14101b54852b3a1c09a86a264a8 in term 1.
I20260927 00:51:12.702137 20217 leader_election.cc:302] T dcce3f5de7954105bb73631074f16ef9 P 6fa9d14101b54852b3a1c09a86a264a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fa9d14101b54852b3a1c09a86a264a8, e80e66aad2284f098ed8b25cdefa1ff9; no voters:
I20260927 00:51:12.702347 20614 raft_consensus.cc:2781] T dcce3f5de7954105bb73631074f16ef9 P 6fa9d14101b54852b3a1c09a86a264a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:12.702615 20614 raft_consensus.cc:686] T dcce3f5de7954105bb73631074f16ef9 P 6fa9d14101b54852b3a1c09a86a264a8 [term 1 LEADER]: Becoming Leader. State: Replica: 6fa9d14101b54852b3a1c09a86a264a8, State: Running, Role: LEADER
I20260927 00:51:12.702965 20614 consensus_queue.cc:237] T dcce3f5de7954105bb73631074f16ef9 P 6fa9d14101b54852b3a1c09a86a264a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa9d14101b54852b3a1c09a86a264a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33831 } } peers { permanent_uuid: "e80e66aad2284f098ed8b25cdefa1ff9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43259 } } peers { permanent_uuid: "d5a3ca67162840399871c5a11b5bfbc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42981 } }
I20260927 00:51:12.706421 20139 catalog_manager.cc:5495] T dcce3f5de7954105bb73631074f16ef9 P 6fa9d14101b54852b3a1c09a86a264a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6fa9d14101b54852b3a1c09a86a264a8 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6fa9d14101b54852b3a1c09a86a264a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fa9d14101b54852b3a1c09a86a264a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e80e66aad2284f098ed8b25cdefa1ff9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5a3ca67162840399871c5a11b5bfbc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42981 } health_report { overall_health: UNKNOWN } } }
I20260927 00:51:12.742710 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:12.738797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55484 (local address 127.0.98.1:33831)
0927 00:51:12.738947 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:12.738965 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:12.738982 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:51:12.739118 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:12.739120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:12.739182 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:12.739301 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:51:12.739585 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.740189 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.741191 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:12.741420 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:12.741531 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:12.741855 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:51:12.742306 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:51:12.742448 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:12.742494 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 00:51:12.742553 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:51:12.748924 20545 raft_consensus.cc:1261] T dcce3f5de7954105bb73631074f16ef9 P e80e66aad2284f098ed8b25cdefa1ff9 [term 1 FOLLOWER]: Refusing update from remote peer 6fa9d14101b54852b3a1c09a86a264a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:51:12.748931 20414 raft_consensus.cc:1261] T dcce3f5de7954105bb73631074f16ef9 P d5a3ca67162840399871c5a11b5bfbc7 [term 1 FOLLOWER]: Refusing update from remote peer 6fa9d14101b54852b3a1c09a86a264a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:51:12.749661 20614 consensus_queue.cc:1035] T dcce3f5de7954105bb73631074f16ef9 P 6fa9d14101b54852b3a1c09a86a264a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5a3ca67162840399871c5a11b5bfbc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:51:12.749886 20619 consensus_queue.cc:1035] T dcce3f5de7954105bb73631074f16ef9 P 6fa9d14101b54852b3a1c09a86a264a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e80e66aad2284f098ed8b25cdefa1ff9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:51:12.757612 20622 mvcc.cc:205] Tried to move back new op lower bound from 7333766237171560448 to 7333766236993998848. Current Snapshot: MvccSnapshot[applied={T|T < 7333766237171560448}]
I20260927 00:51:12.760680 20623 mvcc.cc:205] Tried to move back new op lower bound from 7333766237171560448 to 7333766236993998848. Current Snapshot: MvccSnapshot[applied={T|T < 7333766237171560448 or (T in {7333766237171560448})}]
I20260927 00:51:12.766815 20139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:51:12.766980 20139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:51:12.769600 20139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 005cb63d21f045e08458c15d450f6aa8: Sending DeleteTablet for 3 replicas of tablet dcce3f5de7954105bb73631074f16ef9
I20260927 00:51:12.770532 20263 tablet_service.cc:1514] Processing DeleteTablet for tablet dcce3f5de7954105bb73631074f16ef9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:51:12 UTC) from {username='slave'} at 127.0.0.1:55474
I20260927 00:51:12.770574 20394 tablet_service.cc:1514] Processing DeleteTablet for tablet dcce3f5de7954105bb73631074f16ef9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:51:12 UTC) from {username='slave'} at 127.0.0.1:46410
I20260927 00:51:12.771103 20633 tablet_replica.cc:330] T dcce3f5de7954105bb73631074f16ef9 P d5a3ca67162840399871c5a11b5bfbc7: stopping tablet replica
I20260927 00:51:12.771168 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 20201
I20260927 00:51:12.771102 20525 tablet_service.cc:1514] Processing DeleteTablet for tablet dcce3f5de7954105bb73631074f16ef9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:51:12 UTC) from {username='slave'} at 127.0.0.1:37452
I20260927 00:51:12.772909 20633 raft_consensus.cc:2229] T dcce3f5de7954105bb73631074f16ef9 P d5a3ca67162840399871c5a11b5bfbc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:51:12.773046 20634 tablet_replica.cc:330] T dcce3f5de7954105bb73631074f16ef9 P e80e66aad2284f098ed8b25cdefa1ff9: stopping tablet replica
I20260927 00:51:12.773320 20633 raft_consensus.cc:2258] T dcce3f5de7954105bb73631074f16ef9 P d5a3ca67162840399871c5a11b5bfbc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:51:12.773392 20634 raft_consensus.cc:2229] T dcce3f5de7954105bb73631074f16ef9 P e80e66aad2284f098ed8b25cdefa1ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:51:12.774134 20634 raft_consensus.cc:2258] T dcce3f5de7954105bb73631074f16ef9 P e80e66aad2284f098ed8b25cdefa1ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:51:12.774835 20633 ts_tablet_manager.cc:1905] T dcce3f5de7954105bb73631074f16ef9 P d5a3ca67162840399871c5a11b5bfbc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:51:12.775509 20634 ts_tablet_manager.cc:1905] T dcce3f5de7954105bb73631074f16ef9 P e80e66aad2284f098ed8b25cdefa1ff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:51:12.777211 20633 ts_tablet_manager.cc:1918] T dcce3f5de7954105bb73631074f16ef9 P d5a3ca67162840399871c5a11b5bfbc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:51:12.777326 20633 log.cc:1196] T dcce3f5de7954105bb73631074f16ef9 P d5a3ca67162840399871c5a11b5bfbc7: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-1/wal/wals/dcce3f5de7954105bb73631074f16ef9
I20260927 00:51:12.777632 20633 ts_tablet_manager.cc:1939] T dcce3f5de7954105bb73631074f16ef9 P d5a3ca67162840399871c5a11b5bfbc7: Deleting consensus metadata
I20260927 00:51:12.778803 20634 ts_tablet_manager.cc:1918] T dcce3f5de7954105bb73631074f16ef9 P e80e66aad2284f098ed8b25cdefa1ff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:51:12.778898 20634 log.cc:1196] T dcce3f5de7954105bb73631074f16ef9 P e80e66aad2284f098ed8b25cdefa1ff9: Deleting WAL directory at /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790470006085796-392-0/minicluster-data/ts-2/wal/wals/dcce3f5de7954105bb73631074f16ef9
I20260927 00:51:12.778945 20126 catalog_manager.cc:4841] TS d5a3ca67162840399871c5a11b5bfbc7 (127.0.98.2:42981): tablet dcce3f5de7954105bb73631074f16ef9 (table test-table [id=76e1cd13387c4fc5a78d817986e150d2]) successfully deleted
W20260927 00:51:12.778970 20125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33831 (error 108)
I20260927 00:51:12.779212 20634 ts_tablet_manager.cc:1939] T dcce3f5de7954105bb73631074f16ef9 P e80e66aad2284f098ed8b25cdefa1ff9: Deleting consensus metadata
I20260927 00:51:12.779220 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 20332
I20260927 00:51:12.780541 20126 catalog_manager.cc:4841] TS e80e66aad2284f098ed8b25cdefa1ff9 (127.0.98.3:43259): tablet dcce3f5de7954105bb73631074f16ef9 (table test-table [id=76e1cd13387c4fc5a78d817986e150d2]) successfully deleted
I20260927 00:51:12.785398 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:12.785076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33831 (local address 127.0.0.1:55498)
0927 00:51:12.785218 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:12.785314 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 00:51:12.785596 20125 catalog_manager.cc:4584] TS 6fa9d14101b54852b3a1c09a86a264a8 (127.0.98.1:33831): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dcce3f5de7954105bb73631074f16ef9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33831: connect: Connection refused (error 111)
I20260927 00:51:12.786295 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 20463
I20260927 00:51:12.791765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 20110
2026-09-27T00:51:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1111 ms)
[----------] 6 tests from EncryptionPolicyTest (10952 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:51:12 dist-test-slave-86rs krb5kdc[20638](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:51:12 dist-test-slave-86rs krb5kdc[20638](info): set up 2 sockets
Sep 27 00:51:12 dist-test-slave-86rs krb5kdc[20638](info): commencing operation
krb5kdc: starting...
W20260927 00:51:13.832619 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:51:13 dist-test-slave-86rs krb5kdc[20638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:51:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:51:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:51:13.960110 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43671
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:14.075100 20651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:14.075429 20651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:14.075476 20651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:14.079777 20651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:14.079871 20651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:14.079891 20651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:14.079908 20651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:14.079924 20651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:14.086812 20651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43671
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:14.087989 20651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:14.089160 20651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:14.098845 20657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:14.098832 20656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:14.098938 20651 server_base.cc:1034] running on GCE node
W20260927 00:51:14.098832 20659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:14.099841 20651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:14.101218 20651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:14.104561 20651 hybrid_clock.cc:648] HybridClock initialized: now 1790470274104525 us; error 43 us; skew 500 ppm
Sep 27 00:51:14 dist-test-slave-86rs krb5kdc[20638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:14.108393 20651 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:51:14.109565 20651 webserver.cc:460] Webserver started at http://127.0.0.1:40995/ using document root <none> and password file <none>
I20260927 00:51:14.110242 20651 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:14.110347 20651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:14.110595 20651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:14.113240 20651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "7b24d08e67ac4004bd4e9fa8c4dfa42e"
format_stamp: "Formatted at 2026-09-27 00:51:14 on dist-test-slave-86rs"
server_key: "b16164ae5fd191185dfa0e90419ab102"
server_key_iv: "4b0e19b8748828110c60b0632ee5394c"
server_key_version: "encryptionkey@0"
I20260927 00:51:14.113886 20651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "7b24d08e67ac4004bd4e9fa8c4dfa42e"
format_stamp: "Formatted at 2026-09-27 00:51:14 on dist-test-slave-86rs"
server_key: "b16164ae5fd191185dfa0e90419ab102"
server_key_iv: "4b0e19b8748828110c60b0632ee5394c"
server_key_version: "encryptionkey@0"
I20260927 00:51:14.118010 20651 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:51:14.120956 20666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:14.122555 20651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:51:14.122741 20651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "7b24d08e67ac4004bd4e9fa8c4dfa42e"
format_stamp: "Formatted at 2026-09-27 00:51:14 on dist-test-slave-86rs"
server_key: "b16164ae5fd191185dfa0e90419ab102"
server_key_iv: "4b0e19b8748828110c60b0632ee5394c"
server_key_version: "encryptionkey@0"
I20260927 00:51:14.122906 20651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:14.151628 20651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:14.155328 20651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:14.155566 20651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:14.163141 20651 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43671
I20260927 00:51:14.163159 20718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43671 every 8 connection(s)
I20260927 00:51:14.164165 20651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:51:14.167266 20719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:14.167717 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 20651
I20260927 00:51:14.167878 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:51:14.168206 392 external_mini_cluster.cc:1428] Setting key 9b4b4e8475fbbb3277d024ba6bb09b28
I20260927 00:51:14.175184 20719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b24d08e67ac4004bd4e9fa8c4dfa42e: Bootstrap starting.
I20260927 00:51:14.178113 20719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b24d08e67ac4004bd4e9fa8c4dfa42e: Neither blocks nor log segments found. Creating new log.
Sep 27 00:51:14 dist-test-slave-86rs krb5kdc[20638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 00:51:14.179126 20719 log.cc:824] T 00000000000000000000000000000000 P 7b24d08e67ac4004bd4e9fa8c4dfa42e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:14.181344 20719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b24d08e67ac4004bd4e9fa8c4dfa42e: No bootstrap required, opened a new log
I20260927 00:51:14.184623 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:14.169820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59263 (local address 127.0.0.1:43671)
0927 00:51:14.170293 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:14.170312 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:14.170357 (+ 45us) server_negotiation.cc:407] Connection header received
0927 00:51:14.171146 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:14.171172 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:14.171657 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:14.172066 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:14.172973 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:14.174485 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:14.175483 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:14.176175 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:14.176498 (+ 323us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:14.179491 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:14.179503 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:14.179517 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:14.179548 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:14.181455 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:14.182091 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:14.182097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:14.182103 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:14.182216 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:14.182564 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:14.182567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:14.182568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:14.182972 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:14.183131 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:14.183492 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 00:51:14.183734 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":139,"threads_started":1}
I20260927 00:51:14.185745 20719 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b24d08e67ac4004bd4e9fa8c4dfa42e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b24d08e67ac4004bd4e9fa8c4dfa42e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43671 } }
I20260927 00:51:14.185988 20719 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b24d08e67ac4004bd4e9fa8c4dfa42e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:14.186060 20719 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b24d08e67ac4004bd4e9fa8c4dfa42e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b24d08e67ac4004bd4e9fa8c4dfa42e, State: Initialized, Role: FOLLOWER
I20260927 00:51:14.186501 20719 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b24d08e67ac4004bd4e9fa8c4dfa42e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b24d08e67ac4004bd4e9fa8c4dfa42e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43671 } }
I20260927 00:51:14.186687 20719 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b24d08e67ac4004bd4e9fa8c4dfa42e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:51:14.186759 20719 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b24d08e67ac4004bd4e9fa8c4dfa42e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:51:14.186860 20719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b24d08e67ac4004bd4e9fa8c4dfa42e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:14.187866 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:51:14.188040 20719 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b24d08e67ac4004bd4e9fa8c4dfa42e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b24d08e67ac4004bd4e9fa8c4dfa42e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43671 } }
I20260927 00:51:14.188444 20719 leader_election.cc:302] T 00000000000000000000000000000000 P 7b24d08e67ac4004bd4e9fa8c4dfa42e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b24d08e67ac4004bd4e9fa8c4dfa42e; no voters:
I20260927 00:51:14.188810 20719 leader_election.cc:288] T 00000000000000000000000000000000 P 7b24d08e67ac4004bd4e9fa8c4dfa42e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:51:14.188895 20724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b24d08e67ac4004bd4e9fa8c4dfa42e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:14.189085 20724 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b24d08e67ac4004bd4e9fa8c4dfa42e [term 1 LEADER]: Becoming Leader. State: Replica: 7b24d08e67ac4004bd4e9fa8c4dfa42e, State: Running, Role: LEADER
I20260927 00:51:14.189402 20724 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b24d08e67ac4004bd4e9fa8c4dfa42e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b24d08e67ac4004bd4e9fa8c4dfa42e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43671 } }
I20260927 00:51:14.190238 20719 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b24d08e67ac4004bd4e9fa8c4dfa42e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:51:14.191207 20726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b24d08e67ac4004bd4e9fa8c4dfa42e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b24d08e67ac4004bd4e9fa8c4dfa42e. Latest consensus state: current_term: 1 leader_uuid: "7b24d08e67ac4004bd4e9fa8c4dfa42e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b24d08e67ac4004bd4e9fa8c4dfa42e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43671 } } }
I20260927 00:51:14.191362 20726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b24d08e67ac4004bd4e9fa8c4dfa42e [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:14.191681 20725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b24d08e67ac4004bd4e9fa8c4dfa42e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b24d08e67ac4004bd4e9fa8c4dfa42e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b24d08e67ac4004bd4e9fa8c4dfa42e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43671 } } }
I20260927 00:51:14.191774 20725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b24d08e67ac4004bd4e9fa8c4dfa42e [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:14.191906 20738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:51:14.195538 20738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:51:14.201315 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:14.190214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43083 (local address 127.0.0.1:43671)
0927 00:51:14.190332 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:14.190336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:14.190702 (+ 366us) server_negotiation.cc:407] Connection header received
0927 00:51:14.192497 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:14.192501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:14.192554 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:14.192650 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:14.193470 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:14.194235 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:14.195430 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:14.195756 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:14.195868 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:14.197658 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:14.197665 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:14.197668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:14.197700 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:14.199265 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:14.199775 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:14.199779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:14.199781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:14.199866 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:14.200208 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:14.200211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:14.200213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:14.200499 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:14.200659 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:14.201042 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 00:51:14.201141 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:51:14.202435 20738 catalog_manager.cc:1347] Generated new cluster ID: 6eb2224fe6c44e48831e54a458512aaa
I20260927 00:51:14.202525 20738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:51:14.219015 20738 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:51:14.220057 20738 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:51:14.227837 20738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b24d08e67ac4004bd4e9fa8c4dfa42e: Generated new TSK 0
I20260927 00:51:14.228714 20738 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:51:14.238956 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 20651
W20260927 00:51:14.244602 20636 connection.cc:535] client connection to 127.0.0.1:43671 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43671): Connection reset by peer (error 104)
2026-09-27T00:51:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:51:14 dist-test-slave-86rs krb5kdc[20750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:51:14 dist-test-slave-86rs krb5kdc[20750](info): set up 2 sockets
Sep 27 00:51:14 dist-test-slave-86rs krb5kdc[20750](info): commencing operation
krb5kdc: starting...
W20260927 00:51:15.288489 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:51:15 dist-test-slave-86rs krb5kdc[20750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:51:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:51:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:51:15.419284 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46225
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:15.532577 20763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:15.532876 20763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:15.532919 20763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:15.536993 20763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:15.537055 20763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:15.537075 20763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:15.537091 20763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:15.537106 20763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:15.542474 20763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46225
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:15.543577 20763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:15.544598 20763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:15.553141 20769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:15.553179 20768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:15.553145 20771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:15.553351 20763 server_base.cc:1034] running on GCE node
I20260927 00:51:15.554046 20763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:15.555150 20763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:15.556304 20763 hybrid_clock.cc:648] HybridClock initialized: now 1790470275556270 us; error 50 us; skew 500 ppm
Sep 27 00:51:15 dist-test-slave-86rs krb5kdc[20750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:15.559196 20763 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:51:15.560173 20763 webserver.cc:460] Webserver started at http://127.0.0.1:36629/ using document root <none> and password file <none>
I20260927 00:51:15.560806 20763 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:15.560902 20763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:15.561146 20763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:15.563349 20763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "f194b8c3fbe54ddc9d49291f2ce00ddd"
format_stamp: "Formatted at 2026-09-27 00:51:15 on dist-test-slave-86rs"
server_key: "3e66d2550f8216b840823cca66aa87f4"
server_key_iv: "e126c3abb078d9df2548380318b42200"
server_key_version: "encryptionkey@0"
I20260927 00:51:15.563970 20763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "f194b8c3fbe54ddc9d49291f2ce00ddd"
format_stamp: "Formatted at 2026-09-27 00:51:15 on dist-test-slave-86rs"
server_key: "3e66d2550f8216b840823cca66aa87f4"
server_key_iv: "e126c3abb078d9df2548380318b42200"
server_key_version: "encryptionkey@0"
I20260927 00:51:15.567902 20763 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:51:15.570618 20778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:15.571857 20763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:51:15.572021 20763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "f194b8c3fbe54ddc9d49291f2ce00ddd"
format_stamp: "Formatted at 2026-09-27 00:51:15 on dist-test-slave-86rs"
server_key: "3e66d2550f8216b840823cca66aa87f4"
server_key_iv: "e126c3abb078d9df2548380318b42200"
server_key_version: "encryptionkey@0"
I20260927 00:51:15.572151 20763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:15.593693 20763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:15.596907 20763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:15.597138 20763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:15.604511 20763 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46225
I20260927 00:51:15.604532 20830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46225 every 8 connection(s)
I20260927 00:51:15.605546 20763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:51:15.606088 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 20763
I20260927 00:51:15.606273 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:51:15.606581 392 external_mini_cluster.cc:1428] Setting key 144cf87f25a83c926aa816e04c80adde
I20260927 00:51:15.608811 20831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:15.615877 20831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f194b8c3fbe54ddc9d49291f2ce00ddd: Bootstrap starting.
I20260927 00:51:15.617141 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:15.608087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54101 (local address 127.0.0.1:46225)
0927 00:51:15.608569 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:15.608589 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:15.608631 (+ 42us) server_negotiation.cc:407] Connection header received
0927 00:51:15.609479 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:15.609504 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:15.609951 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:15.610324 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:15.610730 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:15.612271 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:15.613200 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:15.613886 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:15.614060 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:15.615014 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:15.615020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:15.615037 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:15.615537 (+ 500us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:15.615610 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:15.615640 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:15.615777 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:15.615979 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 00:51:15.616119 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":180,"threads_started":1}
I20260927 00:51:15.618702 20831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f194b8c3fbe54ddc9d49291f2ce00ddd: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:15.619578 20831 log.cc:824] T 00000000000000000000000000000000 P f194b8c3fbe54ddc9d49291f2ce00ddd: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:15.620694 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:51:15.621814 20831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f194b8c3fbe54ddc9d49291f2ce00ddd: No bootstrap required, opened a new log
I20260927 00:51:15.625746 20831 raft_consensus.cc:348] T 00000000000000000000000000000000 P f194b8c3fbe54ddc9d49291f2ce00ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f194b8c3fbe54ddc9d49291f2ce00ddd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46225 } }
I20260927 00:51:15.626092 20831 raft_consensus.cc:374] T 00000000000000000000000000000000 P f194b8c3fbe54ddc9d49291f2ce00ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:15.626183 20831 raft_consensus.cc:729] T 00000000000000000000000000000000 P f194b8c3fbe54ddc9d49291f2ce00ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f194b8c3fbe54ddc9d49291f2ce00ddd, State: Initialized, Role: FOLLOWER
I20260927 00:51:15.626727 20831 consensus_queue.cc:260] T 00000000000000000000000000000000 P f194b8c3fbe54ddc9d49291f2ce00ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f194b8c3fbe54ddc9d49291f2ce00ddd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46225 } }
I20260927 00:51:15.626902 20831 raft_consensus.cc:388] T 00000000000000000000000000000000 P f194b8c3fbe54ddc9d49291f2ce00ddd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:51:15.626981 20831 raft_consensus.cc:482] T 00000000000000000000000000000000 P f194b8c3fbe54ddc9d49291f2ce00ddd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:51:15.627081 20831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f194b8c3fbe54ddc9d49291f2ce00ddd [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:15.627753 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:15.622927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39377 (local address 127.0.0.1:46225)
0927 00:51:15.623068 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:15.623075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:15.623197 (+ 122us) server_negotiation.cc:407] Connection header received
0927 00:51:15.623317 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:15.623320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:15.623357 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:15.623427 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:15.623718 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:15.624784 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:15.625946 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:15.626262 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:15.626367 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:15.627300 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:15.627307 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:15.627313 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:15.627330 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:15.627362 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:15.627375 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:15.627479 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:15.627578 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 00:51:15.627619 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 00:51:15.628459 20831 raft_consensus.cc:504] T 00000000000000000000000000000000 P f194b8c3fbe54ddc9d49291f2ce00ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f194b8c3fbe54ddc9d49291f2ce00ddd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46225 } }
I20260927 00:51:15.628829 20831 leader_election.cc:302] T 00000000000000000000000000000000 P f194b8c3fbe54ddc9d49291f2ce00ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f194b8c3fbe54ddc9d49291f2ce00ddd; no voters:
I20260927 00:51:15.629199 20831 leader_election.cc:288] T 00000000000000000000000000000000 P f194b8c3fbe54ddc9d49291f2ce00ddd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:51:15.629317 20841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f194b8c3fbe54ddc9d49291f2ce00ddd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:15.629588 20841 raft_consensus.cc:686] T 00000000000000000000000000000000 P f194b8c3fbe54ddc9d49291f2ce00ddd [term 1 LEADER]: Becoming Leader. State: Replica: f194b8c3fbe54ddc9d49291f2ce00ddd, State: Running, Role: LEADER
I20260927 00:51:15.629935 20841 consensus_queue.cc:237] T 00000000000000000000000000000000 P f194b8c3fbe54ddc9d49291f2ce00ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f194b8c3fbe54ddc9d49291f2ce00ddd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46225 } }
I20260927 00:51:15.630595 20831 sys_catalog.cc:564] T 00000000000000000000000000000000 P f194b8c3fbe54ddc9d49291f2ce00ddd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:51:15.631830 20842 sys_catalog.cc:455] T 00000000000000000000000000000000 P f194b8c3fbe54ddc9d49291f2ce00ddd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f194b8c3fbe54ddc9d49291f2ce00ddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f194b8c3fbe54ddc9d49291f2ce00ddd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46225 } } }
I20260927 00:51:15.631846 20843 sys_catalog.cc:455] T 00000000000000000000000000000000 P f194b8c3fbe54ddc9d49291f2ce00ddd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f194b8c3fbe54ddc9d49291f2ce00ddd. Latest consensus state: current_term: 1 leader_uuid: "f194b8c3fbe54ddc9d49291f2ce00ddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f194b8c3fbe54ddc9d49291f2ce00ddd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46225 } } }
I20260927 00:51:15.631987 20843 sys_catalog.cc:458] T 00000000000000000000000000000000 P f194b8c3fbe54ddc9d49291f2ce00ddd [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:15.631987 20842 sys_catalog.cc:458] T 00000000000000000000000000000000 P f194b8c3fbe54ddc9d49291f2ce00ddd [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:15.632517 20850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:51:15.635454 20850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:51:15.641362 20850 catalog_manager.cc:1347] Generated new cluster ID: 6ea125c77c23467cb9b873a84c5f257b
I20260927 00:51:15.641470 20850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:51:15.661525 20850 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:51:15.662988 20850 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:51:15.669483 20850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f194b8c3fbe54ddc9d49291f2ce00ddd: Generated new TSK 0
I20260927 00:51:15.670377 20850 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:51:15.681895 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 20763
W20260927 00:51:15.687748 20748 connection.cc:535] client connection to 127.0.0.1:46225 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46225): Connection reset by peer (error 104)
2026-09-27T00:51:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:51:15 dist-test-slave-86rs krb5kdc[20862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:51:15 dist-test-slave-86rs krb5kdc[20862](info): set up 2 sockets
Sep 27 00:51:15 dist-test-slave-86rs krb5kdc[20862](info): commencing operation
krb5kdc: starting...
W20260927 00:51:16.724380 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:51:16 dist-test-slave-86rs krb5kdc[20862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:51:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:51:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:51:16.851528 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40629
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:16.964525 20875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:16.964856 20875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:16.964931 20875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:16.969079 20875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:16.969192 20875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:16.969233 20875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:16.969295 20875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:16.969349 20875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:16.974970 20875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40629
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:16.976184 20875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:16.977325 20875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:16.986445 20880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:16.986550 20875 server_base.cc:1034] running on GCE node
W20260927 00:51:16.986469 20883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:16.986457 20881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:16.987327 20875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:16.988648 20875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:16.989838 20875 hybrid_clock.cc:648] HybridClock initialized: now 1790470276989822 us; error 29 us; skew 500 ppm
Sep 27 00:51:16 dist-test-slave-86rs krb5kdc[20862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:16.992880 20875 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:51:16.993872 20875 webserver.cc:460] Webserver started at http://127.0.0.1:37449/ using document root <none> and password file <none>
I20260927 00:51:16.994436 20875 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:16.994498 20875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:16.994674 20875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:16.996912 20875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "827ac3d315be40e4b33b9c61feaa1367"
format_stamp: "Formatted at 2026-09-27 00:51:16 on dist-test-slave-86rs"
server_key: "13eaf84122bb860e7abe4e3937232925"
server_key_iv: "821ff7c7c33485450a9b5a60008322b1"
server_key_version: "encryptionkey@0"
I20260927 00:51:16.997452 20875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "827ac3d315be40e4b33b9c61feaa1367"
format_stamp: "Formatted at 2026-09-27 00:51:16 on dist-test-slave-86rs"
server_key: "13eaf84122bb860e7abe4e3937232925"
server_key_iv: "821ff7c7c33485450a9b5a60008322b1"
server_key_version: "encryptionkey@0"
I20260927 00:51:17.001526 20875 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:51:17.004151 20890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:17.005586 20875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:51:17.005788 20875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "827ac3d315be40e4b33b9c61feaa1367"
format_stamp: "Formatted at 2026-09-27 00:51:16 on dist-test-slave-86rs"
server_key: "13eaf84122bb860e7abe4e3937232925"
server_key_iv: "821ff7c7c33485450a9b5a60008322b1"
server_key_version: "encryptionkey@0"
I20260927 00:51:17.005946 20875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:17.011473 20875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:17.014909 20875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:17.015161 20875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:17.022505 20875 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40629
I20260927 00:51:17.022516 20942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40629 every 8 connection(s)
I20260927 00:51:17.023541 20875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:51:17.026477 20943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:17.028023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 20875
I20260927 00:51:17.028177 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:51:17.028540 392 external_mini_cluster.cc:1428] Setting key 39c0d26b0891ac24509464131d09030f
I20260927 00:51:17.033051 20943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 827ac3d315be40e4b33b9c61feaa1367: Bootstrap starting.
I20260927 00:51:17.034011 20946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:17.030103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38681 (local address 127.0.0.1:40629)
0927 00:51:17.030550 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:17.030565 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:17.030602 (+ 37us) server_negotiation.cc:407] Connection header received
0927 00:51:17.031246 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:17.031263 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:17.031571 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:17.031937 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:17.032764 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:17.032770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:17.032795 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:17.032806 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:17.032858 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:17.032877 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:17.033036 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:17.033107 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 00:51:17.033269 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":134,"threads_started":1}
I20260927 00:51:17.035365 20943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 827ac3d315be40e4b33b9c61feaa1367: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:17.036185 20943 log.cc:824] T 00000000000000000000000000000000 P 827ac3d315be40e4b33b9c61feaa1367: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:17.036883 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:51:17.038635 20943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 827ac3d315be40e4b33b9c61feaa1367: No bootstrap required, opened a new log
I20260927 00:51:17.041082 20946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:17.039318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52993 (local address 127.0.0.1:40629)
0927 00:51:17.039454 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:17.039459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:17.039660 (+ 201us) server_negotiation.cc:407] Connection header received
0927 00:51:17.039794 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:17.039796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:17.039826 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:17.039885 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:17.040656 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:17.040659 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:17.040661 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:17.040671 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:17.040694 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:17.040704 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:17.040775 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:17.040885 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 00:51:17.040918 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:51:17.043340 20943 raft_consensus.cc:348] T 00000000000000000000000000000000 P 827ac3d315be40e4b33b9c61feaa1367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827ac3d315be40e4b33b9c61feaa1367" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40629 } }
I20260927 00:51:17.043627 20943 raft_consensus.cc:374] T 00000000000000000000000000000000 P 827ac3d315be40e4b33b9c61feaa1367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:17.043687 20943 raft_consensus.cc:729] T 00000000000000000000000000000000 P 827ac3d315be40e4b33b9c61feaa1367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 827ac3d315be40e4b33b9c61feaa1367, State: Initialized, Role: FOLLOWER
I20260927 00:51:17.044123 20943 consensus_queue.cc:260] T 00000000000000000000000000000000 P 827ac3d315be40e4b33b9c61feaa1367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827ac3d315be40e4b33b9c61feaa1367" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40629 } }
I20260927 00:51:17.044312 20943 raft_consensus.cc:388] T 00000000000000000000000000000000 P 827ac3d315be40e4b33b9c61feaa1367 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:51:17.044409 20943 raft_consensus.cc:482] T 00000000000000000000000000000000 P 827ac3d315be40e4b33b9c61feaa1367 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:51:17.044513 20943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 827ac3d315be40e4b33b9c61feaa1367 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:17.045629 20943 raft_consensus.cc:504] T 00000000000000000000000000000000 P 827ac3d315be40e4b33b9c61feaa1367 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827ac3d315be40e4b33b9c61feaa1367" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40629 } }
I20260927 00:51:17.046010 20943 leader_election.cc:302] T 00000000000000000000000000000000 P 827ac3d315be40e4b33b9c61feaa1367 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 827ac3d315be40e4b33b9c61feaa1367; no voters:
I20260927 00:51:17.046346 20943 leader_election.cc:288] T 00000000000000000000000000000000 P 827ac3d315be40e4b33b9c61feaa1367 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:51:17.046481 20953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 827ac3d315be40e4b33b9c61feaa1367 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:17.046725 20953 raft_consensus.cc:686] T 00000000000000000000000000000000 P 827ac3d315be40e4b33b9c61feaa1367 [term 1 LEADER]: Becoming Leader. State: Replica: 827ac3d315be40e4b33b9c61feaa1367, State: Running, Role: LEADER
I20260927 00:51:17.047132 20953 consensus_queue.cc:237] T 00000000000000000000000000000000 P 827ac3d315be40e4b33b9c61feaa1367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827ac3d315be40e4b33b9c61feaa1367" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40629 } }
I20260927 00:51:17.047691 20943 sys_catalog.cc:564] T 00000000000000000000000000000000 P 827ac3d315be40e4b33b9c61feaa1367 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:51:17.048804 20954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827ac3d315be40e4b33b9c61feaa1367 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "827ac3d315be40e4b33b9c61feaa1367" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827ac3d315be40e4b33b9c61feaa1367" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40629 } } }
I20260927 00:51:17.048936 20954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827ac3d315be40e4b33b9c61feaa1367 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:17.049227 20955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827ac3d315be40e4b33b9c61feaa1367 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 827ac3d315be40e4b33b9c61feaa1367. Latest consensus state: current_term: 1 leader_uuid: "827ac3d315be40e4b33b9c61feaa1367" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827ac3d315be40e4b33b9c61feaa1367" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40629 } } }
I20260927 00:51:17.049322 20962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:51:17.049346 20955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827ac3d315be40e4b33b9c61feaa1367 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:17.052182 20962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:51:17.057646 20962 catalog_manager.cc:1347] Generated new cluster ID: baec9e424dd24c1085b730888559663e
I20260927 00:51:17.057806 20962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:51:17.080682 20962 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:51:17.081103 20962 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:51:17.089038 20962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 827ac3d315be40e4b33b9c61feaa1367: Generated new TSK 0
I20260927 00:51:17.090018 20962 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:51:17.097898 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 20875
2026-09-27T00:51:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:51:17 dist-test-slave-86rs krb5kdc[20974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:51:17 dist-test-slave-86rs krb5kdc[20974](info): set up 2 sockets
Sep 27 00:51:17 dist-test-slave-86rs krb5kdc[20974](info): commencing operation
krb5kdc: starting...
W20260927 00:51:18.132822 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:51:18 dist-test-slave-86rs krb5kdc[20974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:51:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:51:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:51:18.259945 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33037
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:18.379467 20987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:18.379832 20987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:18.379920 20987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:18.384085 20987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:18.384212 20987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:18.384274 20987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:18.384316 20987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:18.384356 20987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:18.390106 20987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:18.391355 20987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:18.392496 20987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:51:18.401919 20987 server_base.cc:1034] running on GCE node
W20260927 00:51:18.401866 20995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:18.401824 20993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:18.401851 20992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:18.402791 20987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:18.404075 20987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:18.405292 20987 hybrid_clock.cc:648] HybridClock initialized: now 1790470278405262 us; error 58 us; skew 500 ppm
Sep 27 00:51:18 dist-test-slave-86rs krb5kdc[20974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:18.413414 20987 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:51:18.414530 20987 webserver.cc:460] Webserver started at http://127.0.0.1:32893/ using document root <none> and password file <none>
I20260927 00:51:18.415144 20987 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:18.415241 20987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:18.415498 20987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:18.417811 20987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "cda2cccb4fcb4df1815cc4941ba9fdd6"
format_stamp: "Formatted at 2026-09-27 00:51:18 on dist-test-slave-86rs"
server_key: "5676eb535daede62f2fdd52a3852f333"
server_key_iv: "1e52a6707ed912c8ddbe0b87d1f26be9"
server_key_version: "encryptionkey@0"
I20260927 00:51:18.418412 20987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "cda2cccb4fcb4df1815cc4941ba9fdd6"
format_stamp: "Formatted at 2026-09-27 00:51:18 on dist-test-slave-86rs"
server_key: "5676eb535daede62f2fdd52a3852f333"
server_key_iv: "1e52a6707ed912c8ddbe0b87d1f26be9"
server_key_version: "encryptionkey@0"
I20260927 00:51:18.422320 20987 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:51:18.424870 21002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:18.426160 20987 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:51:18.426319 20987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "cda2cccb4fcb4df1815cc4941ba9fdd6"
format_stamp: "Formatted at 2026-09-27 00:51:18 on dist-test-slave-86rs"
server_key: "5676eb535daede62f2fdd52a3852f333"
server_key_iv: "1e52a6707ed912c8ddbe0b87d1f26be9"
server_key_version: "encryptionkey@0"
I20260927 00:51:18.426451 20987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:18.435838 20987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:18.439435 20987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:18.439656 20987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:18.447007 20987 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33037
I20260927 00:51:18.447021 21054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33037 every 8 connection(s)
I20260927 00:51:18.448056 20987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:51:18.451128 21055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:18.456808 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 20987
I20260927 00:51:18.457006 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:51:18.457283 392 external_mini_cluster.cc:1428] Setting key 7c5cc1797784f448d8d7ff001278d919
I20260927 00:51:18.457322 21055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda2cccb4fcb4df1815cc4941ba9fdd6: Bootstrap starting.
I20260927 00:51:18.460105 21055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cda2cccb4fcb4df1815cc4941ba9fdd6: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:18.461133 21055 log.cc:824] T 00000000000000000000000000000000 P cda2cccb4fcb4df1815cc4941ba9fdd6: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:18.463599 21055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda2cccb4fcb4df1815cc4941ba9fdd6: No bootstrap required, opened a new log
Sep 27 00:51:18 dist-test-slave-86rs krb5kdc[20974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 00:51:18.468783 21055 raft_consensus.cc:348] T 00000000000000000000000000000000 P cda2cccb4fcb4df1815cc4941ba9fdd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda2cccb4fcb4df1815cc4941ba9fdd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33037 } }
I20260927 00:51:18.469087 21055 raft_consensus.cc:374] T 00000000000000000000000000000000 P cda2cccb4fcb4df1815cc4941ba9fdd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:18.469172 21055 raft_consensus.cc:729] T 00000000000000000000000000000000 P cda2cccb4fcb4df1815cc4941ba9fdd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda2cccb4fcb4df1815cc4941ba9fdd6, State: Initialized, Role: FOLLOWER
I20260927 00:51:18.469777 21055 consensus_queue.cc:260] T 00000000000000000000000000000000 P cda2cccb4fcb4df1815cc4941ba9fdd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda2cccb4fcb4df1815cc4941ba9fdd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33037 } }
I20260927 00:51:18.469933 21055 raft_consensus.cc:388] T 00000000000000000000000000000000 P cda2cccb4fcb4df1815cc4941ba9fdd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:51:18.470058 21055 raft_consensus.cc:482] T 00000000000000000000000000000000 P cda2cccb4fcb4df1815cc4941ba9fdd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:51:18.470204 21055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cda2cccb4fcb4df1815cc4941ba9fdd6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:18.471243 21055 raft_consensus.cc:504] T 00000000000000000000000000000000 P cda2cccb4fcb4df1815cc4941ba9fdd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda2cccb4fcb4df1815cc4941ba9fdd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33037 } }
I20260927 00:51:18.471664 21055 leader_election.cc:302] T 00000000000000000000000000000000 P cda2cccb4fcb4df1815cc4941ba9fdd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cda2cccb4fcb4df1815cc4941ba9fdd6; no voters:
I20260927 00:51:18.472064 21055 leader_election.cc:288] T 00000000000000000000000000000000 P cda2cccb4fcb4df1815cc4941ba9fdd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:51:18.472175 21060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cda2cccb4fcb4df1815cc4941ba9fdd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:18.472469 21060 raft_consensus.cc:686] T 00000000000000000000000000000000 P cda2cccb4fcb4df1815cc4941ba9fdd6 [term 1 LEADER]: Becoming Leader. State: Replica: cda2cccb4fcb4df1815cc4941ba9fdd6, State: Running, Role: LEADER
I20260927 00:51:18.472877 21060 consensus_queue.cc:237] T 00000000000000000000000000000000 P cda2cccb4fcb4df1815cc4941ba9fdd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda2cccb4fcb4df1815cc4941ba9fdd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33037 } }
I20260927 00:51:18.473515 21055 sys_catalog.cc:564] T 00000000000000000000000000000000 P cda2cccb4fcb4df1815cc4941ba9fdd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:51:18.474447 21058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:18.458669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52105 (local address 127.0.0.1:33037)
0927 00:51:18.459117 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:18.459137 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:18.459174 (+ 37us) server_negotiation.cc:407] Connection header received
0927 00:51:18.459891 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:18.459912 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:18.460363 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:18.460687 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:18.461698 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:18.462861 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:18.464201 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:18.464788 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:18.464927 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:18.468183 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:18.468190 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:18.468208 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:18.468303 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:18.470610 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:18.471133 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:18.471140 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:18.471147 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:18.471281 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:18.471666 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:18.471668 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:18.471670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:18.472355 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:18.472553 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:18.473043 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 00:51:18.473380 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":161,"threads_started":1}
I20260927 00:51:18.475445 21062 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda2cccb4fcb4df1815cc4941ba9fdd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cda2cccb4fcb4df1815cc4941ba9fdd6. Latest consensus state: current_term: 1 leader_uuid: "cda2cccb4fcb4df1815cc4941ba9fdd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda2cccb4fcb4df1815cc4941ba9fdd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33037 } } }
I20260927 00:51:18.475570 21062 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda2cccb4fcb4df1815cc4941ba9fdd6 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:18.475591 21061 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda2cccb4fcb4df1815cc4941ba9fdd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cda2cccb4fcb4df1815cc4941ba9fdd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda2cccb4fcb4df1815cc4941ba9fdd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33037 } } }
I20260927 00:51:18.475697 21061 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda2cccb4fcb4df1815cc4941ba9fdd6 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:18.475944 21074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:51:18.478288 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:51:18.479799 21074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:51:18.486992 21074 catalog_manager.cc:1347] Generated new cluster ID: 3b4089467e1a4a8caeb33d6c06ea6e12
I20260927 00:51:18.487074 21074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:51:18.489742 21058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:18.480670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52671 (local address 127.0.0.1:33037)
0927 00:51:18.480792 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:18.480796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:18.481043 (+ 247us) server_negotiation.cc:407] Connection header received
0927 00:51:18.481179 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:18.481182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:18.481238 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:18.481320 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:18.482201 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:18.482733 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:18.483586 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:18.483853 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:18.483972 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:18.486037 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:18.486042 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:18.486045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:18.486074 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:18.487907 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:18.488448 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:18.488452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:18.488454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:18.488552 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:18.488827 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:18.488830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:18.488832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:18.489069 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:18.489202 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:18.489436 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 00:51:18.489565 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:51:18.495081 21074 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:51:18.496398 21074 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:51:18.506913 21074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cda2cccb4fcb4df1815cc4941ba9fdd6: Generated new TSK 0
I20260927 00:51:18.507750 21074 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:51:18.516201 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 20987
2026-09-27T00:51:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1417 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:51:18 dist-test-slave-86rs krb5kdc[21086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:51:18 dist-test-slave-86rs krb5kdc[21086](info): set up 2 sockets
Sep 27 00:51:18 dist-test-slave-86rs krb5kdc[21086](info): commencing operation
krb5kdc: starting...
W20260927 00:51:19.552547 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:51:19 dist-test-slave-86rs krb5kdc[21086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:51:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:51:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:51:19.681612 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36643
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:19.797179 21099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:19.797485 21099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:19.797570 21099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:19.801899 21099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:19.801985 21099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:19.802004 21099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:19.802021 21099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:19.802037 21099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:19.807845 21099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:19.809119 21099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:19.810220 21099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:19.819291 21104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:19.819329 21107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:19.819352 21105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:19.819564 21099 server_base.cc:1034] running on GCE node
I20260927 00:51:19.820315 21099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:19.821576 21099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:19.822746 21099 hybrid_clock.cc:648] HybridClock initialized: now 1790470279822720 us; error 40 us; skew 500 ppm
Sep 27 00:51:19 dist-test-slave-86rs krb5kdc[21086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:19.825785 21099 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:51:19.826753 21099 webserver.cc:460] Webserver started at http://127.0.0.1:33337/ using document root <none> and password file <none>
I20260927 00:51:19.827368 21099 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:19.827463 21099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:19.827708 21099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:19.829968 21099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "7d2b1e9087cf49f684b89ec0d544eb75"
format_stamp: "Formatted at 2026-09-27 00:51:19 on dist-test-slave-86rs"
server_key: "bbed3de0e8f77c91050aba364992578f"
server_key_iv: "f1558b8593fa678aad0cd523baa4cdc2"
server_key_version: "encryptionkey@0"
I20260927 00:51:19.830571 21099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "7d2b1e9087cf49f684b89ec0d544eb75"
format_stamp: "Formatted at 2026-09-27 00:51:19 on dist-test-slave-86rs"
server_key: "bbed3de0e8f77c91050aba364992578f"
server_key_iv: "f1558b8593fa678aad0cd523baa4cdc2"
server_key_version: "encryptionkey@0"
I20260927 00:51:19.834589 21099 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:51:19.837361 21114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:19.838682 21099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:51:19.838832 21099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "7d2b1e9087cf49f684b89ec0d544eb75"
format_stamp: "Formatted at 2026-09-27 00:51:19 on dist-test-slave-86rs"
server_key: "bbed3de0e8f77c91050aba364992578f"
server_key_iv: "f1558b8593fa678aad0cd523baa4cdc2"
server_key_version: "encryptionkey@0"
I20260927 00:51:19.838975 21099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:19.861857 21099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:19.865373 21099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:19.865602 21099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:19.873016 21099 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36643
I20260927 00:51:19.873039 21166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36643 every 8 connection(s)
I20260927 00:51:19.874042 21099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:51:19.876959 21167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:19.879041 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 21099
I20260927 00:51:19.879197 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:51:19.879516 392 external_mini_cluster.cc:1428] Setting key 91c717cac2dd56bb2f20901c63b87da5
I20260927 00:51:19.884168 21167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d2b1e9087cf49f684b89ec0d544eb75: Bootstrap starting.
I20260927 00:51:19.886628 21167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d2b1e9087cf49f684b89ec0d544eb75: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:19.887562 21167 log.cc:824] T 00000000000000000000000000000000 P 7d2b1e9087cf49f684b89ec0d544eb75: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:19.889418 21170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:19.881165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36883 (local address 127.0.0.1:36643)
0927 00:51:19.881688 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:19.881704 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:19.881744 (+ 40us) server_negotiation.cc:407] Connection header received
0927 00:51:19.882475 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:19.882495 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:19.882907 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:19.883287 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:19.883724 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:19.884882 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:19.885843 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:19.886334 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:19.886513 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:19.887492 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:19.887497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:19.887511 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:19.887992 (+ 481us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:19.888064 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:19.888095 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:19.888197 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:19.888418 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 00:51:19.888661 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":166,"threads_started":1}
I20260927 00:51:19.889827 21167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d2b1e9087cf49f684b89ec0d544eb75: No bootstrap required, opened a new log
I20260927 00:51:19.892475 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:51:19.894639 21167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d2b1e9087cf49f684b89ec0d544eb75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d2b1e9087cf49f684b89ec0d544eb75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36643 } }
I20260927 00:51:19.894987 21167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d2b1e9087cf49f684b89ec0d544eb75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:19.895079 21167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d2b1e9087cf49f684b89ec0d544eb75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d2b1e9087cf49f684b89ec0d544eb75, State: Initialized, Role: FOLLOWER
I20260927 00:51:19.895592 21167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d2b1e9087cf49f684b89ec0d544eb75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d2b1e9087cf49f684b89ec0d544eb75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36643 } }
I20260927 00:51:19.895749 21167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d2b1e9087cf49f684b89ec0d544eb75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:51:19.895851 21167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d2b1e9087cf49f684b89ec0d544eb75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:51:19.896008 21167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d2b1e9087cf49f684b89ec0d544eb75 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:19.897228 21167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d2b1e9087cf49f684b89ec0d544eb75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d2b1e9087cf49f684b89ec0d544eb75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36643 } }
I20260927 00:51:19.897596 21167 leader_election.cc:302] T 00000000000000000000000000000000 P 7d2b1e9087cf49f684b89ec0d544eb75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d2b1e9087cf49f684b89ec0d544eb75; no voters:
I20260927 00:51:19.897961 21167 leader_election.cc:288] T 00000000000000000000000000000000 P 7d2b1e9087cf49f684b89ec0d544eb75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:51:19.898026 21177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d2b1e9087cf49f684b89ec0d544eb75 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:19.898249 21177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d2b1e9087cf49f684b89ec0d544eb75 [term 1 LEADER]: Becoming Leader. State: Replica: 7d2b1e9087cf49f684b89ec0d544eb75, State: Running, Role: LEADER
I20260927 00:51:19.898967 21177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d2b1e9087cf49f684b89ec0d544eb75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d2b1e9087cf49f684b89ec0d544eb75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36643 } }
I20260927 00:51:19.899451 21167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d2b1e9087cf49f684b89ec0d544eb75 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:51:19.899509 21170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:19.894960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52855 (local address 127.0.0.1:36643)
0927 00:51:19.895089 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:19.895095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:19.895379 (+ 284us) server_negotiation.cc:407] Connection header received
0927 00:51:19.895552 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:19.895555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:19.895598 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:19.895730 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:19.896142 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:19.896730 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:19.897648 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:19.898011 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:19.898121 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:19.899084 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:19.899090 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:19.899093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:19.899106 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:19.899135 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:19.899148 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:19.899208 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:19.899297 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 00:51:19.899362 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:51:19.901177 21178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d2b1e9087cf49f684b89ec0d544eb75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d2b1e9087cf49f684b89ec0d544eb75. Latest consensus state: current_term: 1 leader_uuid: "7d2b1e9087cf49f684b89ec0d544eb75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d2b1e9087cf49f684b89ec0d544eb75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36643 } } }
I20260927 00:51:19.901212 21179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d2b1e9087cf49f684b89ec0d544eb75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d2b1e9087cf49f684b89ec0d544eb75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d2b1e9087cf49f684b89ec0d544eb75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36643 } } }
I20260927 00:51:19.901330 21179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d2b1e9087cf49f684b89ec0d544eb75 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:19.901316 21178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d2b1e9087cf49f684b89ec0d544eb75 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:19.901789 21186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:51:19.905699 21186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:51:19.911623 21186 catalog_manager.cc:1347] Generated new cluster ID: 18b9c1891c454228ba796a9fabd3a86a
I20260927 00:51:19.911706 21186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:51:19.935992 21186 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:51:19.937227 21186 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:51:19.948678 21186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d2b1e9087cf49f684b89ec0d544eb75: Generated new TSK 0
I20260927 00:51:19.949489 21186 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:51:19.957865 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 21099
2026-09-27T00:51:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1442 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:51:20 dist-test-slave-86rs krb5kdc[21198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:51:20 dist-test-slave-86rs krb5kdc[21198](info): set up 2 sockets
Sep 27 00:51:20 dist-test-slave-86rs krb5kdc[21198](info): commencing operation
krb5kdc: starting...
W20260927 00:51:20.992563 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:51:21 dist-test-slave-86rs krb5kdc[21198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:51:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:51:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:51:21.127241 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40719
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:21.243194 21211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:21.243557 21211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:21.243643 21211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:21.247913 21211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:21.248037 21211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:21.248077 21211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:21.248133 21211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:21.248170 21211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:21.253897 21211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:21.255095 21211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:21.256171 21211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:21.265069 21217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:21.265172 21219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:21.265188 21216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:21.265470 21211 server_base.cc:1034] running on GCE node
I20260927 00:51:21.266083 21211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:21.267352 21211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:21.268584 21211 hybrid_clock.cc:648] HybridClock initialized: now 1790470281268572 us; error 36 us; skew 500 ppm
Sep 27 00:51:21 dist-test-slave-86rs krb5kdc[21198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:21.271634 21211 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:51:21.272678 21211 webserver.cc:460] Webserver started at http://127.0.0.1:45997/ using document root <none> and password file <none>
I20260927 00:51:21.273289 21211 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:21.273386 21211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:21.273633 21211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:21.275767 21211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "dddd72b9789c4a25b2f30d0627fd5f71"
format_stamp: "Formatted at 2026-09-27 00:51:21 on dist-test-slave-86rs"
server_key: "d30d8378ba5345a7217524cd00f8cf43"
server_key_iv: "e8d3491dbc9739ca95661cc10fc72c6c"
server_key_version: "encryptionkey@0"
I20260927 00:51:21.276434 21211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "dddd72b9789c4a25b2f30d0627fd5f71"
format_stamp: "Formatted at 2026-09-27 00:51:21 on dist-test-slave-86rs"
server_key: "d30d8378ba5345a7217524cd00f8cf43"
server_key_iv: "e8d3491dbc9739ca95661cc10fc72c6c"
server_key_version: "encryptionkey@0"
I20260927 00:51:21.280486 21211 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:51:21.283026 21226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:21.284376 21211 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:51:21.284519 21211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "dddd72b9789c4a25b2f30d0627fd5f71"
format_stamp: "Formatted at 2026-09-27 00:51:21 on dist-test-slave-86rs"
server_key: "d30d8378ba5345a7217524cd00f8cf43"
server_key_iv: "e8d3491dbc9739ca95661cc10fc72c6c"
server_key_version: "encryptionkey@0"
I20260927 00:51:21.284649 21211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:21.289459 21211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:21.292867 21211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:21.293084 21211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:21.300141 21211 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40719
I20260927 00:51:21.300190 21278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40719 every 8 connection(s)
I20260927 00:51:21.301244 21211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:51:21.304172 21279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:21.304194 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 21211
I20260927 00:51:21.304396 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:51:21.304706 392 external_mini_cluster.cc:1428] Setting key f927a95290796f8d0b5f0ee72ad2e569
I20260927 00:51:21.310781 21282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:21.306146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34349 (local address 127.0.0.1:40719)
0927 00:51:21.306575 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:21.306594 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:21.306637 (+ 43us) server_negotiation.cc:407] Connection header received
0927 00:51:21.307746 (+ 1109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:21.307772 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:21.308162 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:21.308575 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:21.309335 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:21.309341 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:21.309356 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:21.309368 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:21.309431 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:21.309457 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:21.309605 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:21.309677 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 00:51:21.309844 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":111,"threads_started":1}
I20260927 00:51:21.311923 21279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dddd72b9789c4a25b2f30d0627fd5f71: Bootstrap starting.
I20260927 00:51:21.313778 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:51:21.314538 21279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dddd72b9789c4a25b2f30d0627fd5f71: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:21.315363 21279 log.cc:824] T 00000000000000000000000000000000 P dddd72b9789c4a25b2f30d0627fd5f71: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:21.317447 21279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dddd72b9789c4a25b2f30d0627fd5f71: No bootstrap required, opened a new log
I20260927 00:51:21.319191 21282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:21.316046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35327 (local address 127.0.0.1:40719)
0927 00:51:21.316183 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:21.316188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:21.317776 (+ 1588us) server_negotiation.cc:407] Connection header received
0927 00:51:21.317940 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:21.317943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:21.317977 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:21.318060 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:21.318803 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:21.318806 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:21.318809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:21.318820 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:21.318845 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:21.318862 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:21.318950 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:21.319030 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 00:51:21.319069 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:51:21.321364 21279 raft_consensus.cc:348] T 00000000000000000000000000000000 P dddd72b9789c4a25b2f30d0627fd5f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dddd72b9789c4a25b2f30d0627fd5f71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40719 } }
I20260927 00:51:21.321593 21279 raft_consensus.cc:374] T 00000000000000000000000000000000 P dddd72b9789c4a25b2f30d0627fd5f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:21.321666 21279 raft_consensus.cc:729] T 00000000000000000000000000000000 P dddd72b9789c4a25b2f30d0627fd5f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dddd72b9789c4a25b2f30d0627fd5f71, State: Initialized, Role: FOLLOWER
I20260927 00:51:21.322144 21279 consensus_queue.cc:260] T 00000000000000000000000000000000 P dddd72b9789c4a25b2f30d0627fd5f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dddd72b9789c4a25b2f30d0627fd5f71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40719 } }
I20260927 00:51:21.322269 21279 raft_consensus.cc:388] T 00000000000000000000000000000000 P dddd72b9789c4a25b2f30d0627fd5f71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:51:21.322340 21279 raft_consensus.cc:482] T 00000000000000000000000000000000 P dddd72b9789c4a25b2f30d0627fd5f71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:51:21.322445 21279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dddd72b9789c4a25b2f30d0627fd5f71 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:21.323411 21279 raft_consensus.cc:504] T 00000000000000000000000000000000 P dddd72b9789c4a25b2f30d0627fd5f71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dddd72b9789c4a25b2f30d0627fd5f71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40719 } }
I20260927 00:51:21.323757 21279 leader_election.cc:302] T 00000000000000000000000000000000 P dddd72b9789c4a25b2f30d0627fd5f71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dddd72b9789c4a25b2f30d0627fd5f71; no voters:
I20260927 00:51:21.324070 21279 leader_election.cc:288] T 00000000000000000000000000000000 P dddd72b9789c4a25b2f30d0627fd5f71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:51:21.324267 21289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dddd72b9789c4a25b2f30d0627fd5f71 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:21.324513 21289 raft_consensus.cc:686] T 00000000000000000000000000000000 P dddd72b9789c4a25b2f30d0627fd5f71 [term 1 LEADER]: Becoming Leader. State: Replica: dddd72b9789c4a25b2f30d0627fd5f71, State: Running, Role: LEADER
I20260927 00:51:21.324894 21289 consensus_queue.cc:237] T 00000000000000000000000000000000 P dddd72b9789c4a25b2f30d0627fd5f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dddd72b9789c4a25b2f30d0627fd5f71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40719 } }
I20260927 00:51:21.325414 21279 sys_catalog.cc:564] T 00000000000000000000000000000000 P dddd72b9789c4a25b2f30d0627fd5f71 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:51:21.326714 21291 sys_catalog.cc:455] T 00000000000000000000000000000000 P dddd72b9789c4a25b2f30d0627fd5f71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dddd72b9789c4a25b2f30d0627fd5f71. Latest consensus state: current_term: 1 leader_uuid: "dddd72b9789c4a25b2f30d0627fd5f71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dddd72b9789c4a25b2f30d0627fd5f71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40719 } } }
I20260927 00:51:21.326723 21290 sys_catalog.cc:455] T 00000000000000000000000000000000 P dddd72b9789c4a25b2f30d0627fd5f71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dddd72b9789c4a25b2f30d0627fd5f71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dddd72b9789c4a25b2f30d0627fd5f71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40719 } } }
I20260927 00:51:21.326845 21291 sys_catalog.cc:458] T 00000000000000000000000000000000 P dddd72b9789c4a25b2f30d0627fd5f71 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:21.326855 21290 sys_catalog.cc:458] T 00000000000000000000000000000000 P dddd72b9789c4a25b2f30d0627fd5f71 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:21.327268 21298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:51:21.330550 21298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:51:21.336715 21298 catalog_manager.cc:1347] Generated new cluster ID: 7894e311af7c4d8e92f52da439ec0e45
I20260927 00:51:21.336822 21298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:51:21.355216 21298 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:51:21.355603 21298 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:51:21.365710 21298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dddd72b9789c4a25b2f30d0627fd5f71: Generated new TSK 0
I20260927 00:51:21.366509 21298 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:51:21.379491 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 21211
2026-09-27T00:51:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:51:21 dist-test-slave-86rs krb5kdc[21310](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:51:21 dist-test-slave-86rs krb5kdc[21310](info): set up 2 sockets
Sep 27 00:51:21 dist-test-slave-86rs krb5kdc[21310](info): commencing operation
krb5kdc: starting...
W20260927 00:51:22.428561 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:51:22 dist-test-slave-86rs krb5kdc[21310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:51:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:51:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:51:22.557750 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45139
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:22.670909 21323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:22.671211 21323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:22.671254 21323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:22.675383 21323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:22.675458 21323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:22.675478 21323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:22.675496 21323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:22.675511 21323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:22.681173 21323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:22.682340 21323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:22.683403 21323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:22.692274 21329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:22.692411 21328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:22.692577 21323 server_base.cc:1034] running on GCE node
W20260927 00:51:22.692411 21331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:22.693310 21323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:22.694550 21323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:22.695741 21323 hybrid_clock.cc:648] HybridClock initialized: now 1790470282695711 us; error 47 us; skew 500 ppm
Sep 27 00:51:22 dist-test-slave-86rs krb5kdc[21310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:22.698710 21323 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:51:22.699806 21323 webserver.cc:460] Webserver started at http://127.0.0.1:43023/ using document root <none> and password file <none>
I20260927 00:51:22.700474 21323 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:22.700541 21323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:22.700766 21323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:22.702962 21323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "a256385ff72a43a4831cc0fc3f99640d"
format_stamp: "Formatted at 2026-09-27 00:51:22 on dist-test-slave-86rs"
server_key: "f03965de90388e3f5f9f54b0e794b4cd"
server_key_iv: "37247cffbc9ce33b38f97a07ef3800ba"
server_key_version: "encryptionkey@0"
I20260927 00:51:22.703570 21323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "a256385ff72a43a4831cc0fc3f99640d"
format_stamp: "Formatted at 2026-09-27 00:51:22 on dist-test-slave-86rs"
server_key: "f03965de90388e3f5f9f54b0e794b4cd"
server_key_iv: "37247cffbc9ce33b38f97a07ef3800ba"
server_key_version: "encryptionkey@0"
I20260927 00:51:22.707530 21323 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:51:22.710211 21338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:22.711510 21323 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 00:51:22.711673 21323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "a256385ff72a43a4831cc0fc3f99640d"
format_stamp: "Formatted at 2026-09-27 00:51:22 on dist-test-slave-86rs"
server_key: "f03965de90388e3f5f9f54b0e794b4cd"
server_key_iv: "37247cffbc9ce33b38f97a07ef3800ba"
server_key_version: "encryptionkey@0"
I20260927 00:51:22.711820 21323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:22.727231 21323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:22.730760 21323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:22.731021 21323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:22.739167 21323 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45139
I20260927 00:51:22.739212 21390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45139 every 8 connection(s)
I20260927 00:51:22.740515 21323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:51:22.743497 21391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:22.744768 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 21323
I20260927 00:51:22.744901 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:51:22.745232 392 external_mini_cluster.cc:1428] Setting key da134ff4ba12a41575b57e9acdbe9ee7
I20260927 00:51:22.749866 21391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a256385ff72a43a4831cc0fc3f99640d: Bootstrap starting.
I20260927 00:51:22.752336 21391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a256385ff72a43a4831cc0fc3f99640d: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:22.753227 21391 log.cc:824] T 00000000000000000000000000000000 P a256385ff72a43a4831cc0fc3f99640d: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:51:22 dist-test-slave-86rs krb5kdc[21310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 00:51:22.755419 21391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a256385ff72a43a4831cc0fc3f99640d: No bootstrap required, opened a new log
I20260927 00:51:22.759297 21391 raft_consensus.cc:348] T 00000000000000000000000000000000 P a256385ff72a43a4831cc0fc3f99640d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a256385ff72a43a4831cc0fc3f99640d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45139 } }
I20260927 00:51:22.759671 21391 raft_consensus.cc:374] T 00000000000000000000000000000000 P a256385ff72a43a4831cc0fc3f99640d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:22.759763 21391 raft_consensus.cc:729] T 00000000000000000000000000000000 P a256385ff72a43a4831cc0fc3f99640d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a256385ff72a43a4831cc0fc3f99640d, State: Initialized, Role: FOLLOWER
I20260927 00:51:22.760356 21391 consensus_queue.cc:260] T 00000000000000000000000000000000 P a256385ff72a43a4831cc0fc3f99640d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a256385ff72a43a4831cc0fc3f99640d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45139 } }
I20260927 00:51:22.760524 21391 raft_consensus.cc:388] T 00000000000000000000000000000000 P a256385ff72a43a4831cc0fc3f99640d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:51:22.760608 21394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:22.746672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:36685 (local address 127.0.0.1:45139)
0927 00:51:22.747141 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:22.747163 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:22.747414 (+ 251us) server_negotiation.cc:407] Connection header received
0927 00:51:22.748197 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:22.748215 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:22.748659 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:22.749007 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:22.749820 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:22.750886 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:22.751798 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:22.752375 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:22.752563 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:22.755417 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:22.755422 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:22.755437 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:22.755465 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:22.757248 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:22.757798 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:22.757804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:22.757809 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:22.757924 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:22.758281 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:22.758291 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:22.758292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:22.758714 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:22.758956 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:22.759299 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 00:51:22.759556 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":150,"threads_started":1}
I20260927 00:51:22.760599 21391 raft_consensus.cc:482] T 00000000000000000000000000000000 P a256385ff72a43a4831cc0fc3f99640d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:51:22.760828 21391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a256385ff72a43a4831cc0fc3f99640d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:22.761919 21391 raft_consensus.cc:504] T 00000000000000000000000000000000 P a256385ff72a43a4831cc0fc3f99640d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a256385ff72a43a4831cc0fc3f99640d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45139 } }
I20260927 00:51:22.762308 21391 leader_election.cc:302] T 00000000000000000000000000000000 P a256385ff72a43a4831cc0fc3f99640d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a256385ff72a43a4831cc0fc3f99640d; no voters:
I20260927 00:51:22.762645 21391 leader_election.cc:288] T 00000000000000000000000000000000 P a256385ff72a43a4831cc0fc3f99640d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:51:22.762774 21396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a256385ff72a43a4831cc0fc3f99640d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:22.762990 21396 raft_consensus.cc:686] T 00000000000000000000000000000000 P a256385ff72a43a4831cc0fc3f99640d [term 1 LEADER]: Becoming Leader. State: Replica: a256385ff72a43a4831cc0fc3f99640d, State: Running, Role: LEADER
I20260927 00:51:22.763396 21396 consensus_queue.cc:237] T 00000000000000000000000000000000 P a256385ff72a43a4831cc0fc3f99640d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a256385ff72a43a4831cc0fc3f99640d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45139 } }
I20260927 00:51:22.764017 21391 sys_catalog.cc:564] T 00000000000000000000000000000000 P a256385ff72a43a4831cc0fc3f99640d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:51:22.764317 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:51:22.766872 21398 sys_catalog.cc:455] T 00000000000000000000000000000000 P a256385ff72a43a4831cc0fc3f99640d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a256385ff72a43a4831cc0fc3f99640d. Latest consensus state: current_term: 1 leader_uuid: "a256385ff72a43a4831cc0fc3f99640d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a256385ff72a43a4831cc0fc3f99640d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45139 } } }
I20260927 00:51:22.767016 21398 sys_catalog.cc:458] T 00000000000000000000000000000000 P a256385ff72a43a4831cc0fc3f99640d [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:22.767377 21416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:51:22.767915 21397 sys_catalog.cc:455] T 00000000000000000000000000000000 P a256385ff72a43a4831cc0fc3f99640d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a256385ff72a43a4831cc0fc3f99640d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a256385ff72a43a4831cc0fc3f99640d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45139 } } }
I20260927 00:51:22.768083 21397 sys_catalog.cc:458] T 00000000000000000000000000000000 P a256385ff72a43a4831cc0fc3f99640d [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:22.770612 21416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:51:22.775060 21394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:22.766728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:46303 (local address 127.0.0.1:45139)
0927 00:51:22.766858 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:22.766864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:22.767072 (+ 208us) server_negotiation.cc:407] Connection header received
0927 00:51:22.767205 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:22.767208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:22.767268 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:22.767366 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:22.768407 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:22.768937 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:22.769767 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:22.770022 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:22.770136 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:22.771619 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:22.771623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:22.771626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:22.771654 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:22.773216 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:22.773710 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:22.773714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:22.773716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:22.773802 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:22.774132 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:22.774135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:22.774137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:22.774324 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:22.774481 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:22.774760 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 00:51:22.774858 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 00:51:22.777558 21416 catalog_manager.cc:1347] Generated new cluster ID: f0a8dfbed6214ef69954896c3e61da7c
I20260927 00:51:22.777642 21416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:51:22.792999 21416 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:51:22.794035 21416 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:51:22.807598 21416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a256385ff72a43a4831cc0fc3f99640d: Generated new TSK 0
I20260927 00:51:22.808393 21416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:51:22.815593 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 21323
W20260927 00:51:22.821062 21308 connection.cc:535] client connection to 127.0.0.1:45139 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45139): Connection reset by peer (error 104)
2026-09-27T00:51:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:51:22 dist-test-slave-86rs krb5kdc[21422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:51:22 dist-test-slave-86rs krb5kdc[21422](info): set up 2 sockets
Sep 27 00:51:22 dist-test-slave-86rs krb5kdc[21422](info): commencing operation
krb5kdc: starting...
W20260927 00:51:23.848763 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:51:23 dist-test-slave-86rs krb5kdc[21422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:51:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:51:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:51:23.977362 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46201
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:24.096503 21435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:24.096841 21435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:24.096887 21435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:24.101652 21435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:24.101761 21435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:24.101784 21435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:24.101807 21435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:24.101823 21435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:24.107662 21435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:24.108955 21435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:24.109997 21435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:24.119943 21443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:24.119977 21441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:24.120075 21435 server_base.cc:1034] running on GCE node
W20260927 00:51:24.119988 21440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:24.120934 21435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:24.122074 21435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:24.123268 21435 hybrid_clock.cc:648] HybridClock initialized: now 1790470284123233 us; error 47 us; skew 500 ppm
Sep 27 00:51:24 dist-test-slave-86rs krb5kdc[21422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:24.132382 21435 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:51:24.133612 21435 webserver.cc:460] Webserver started at http://127.0.0.1:42493/ using document root <none> and password file <none>
I20260927 00:51:24.134275 21435 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:24.134380 21435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:24.134636 21435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:24.137185 21435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "946e02278c8e42e1967f5c0e440c44a6"
format_stamp: "Formatted at 2026-09-27 00:51:24 on dist-test-slave-86rs"
server_key: "ae380e24b9fad5ed4053beedd74fb362"
server_key_iv: "5a8d06244b29aebbeb593115b0cb9ec5"
server_key_version: "encryptionkey@0"
I20260927 00:51:24.137847 21435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "946e02278c8e42e1967f5c0e440c44a6"
format_stamp: "Formatted at 2026-09-27 00:51:24 on dist-test-slave-86rs"
server_key: "ae380e24b9fad5ed4053beedd74fb362"
server_key_iv: "5a8d06244b29aebbeb593115b0cb9ec5"
server_key_version: "encryptionkey@0"
I20260927 00:51:24.142212 21435 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 00:51:24.145201 21450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:24.146783 21435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:51:24.146984 21435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "946e02278c8e42e1967f5c0e440c44a6"
format_stamp: "Formatted at 2026-09-27 00:51:24 on dist-test-slave-86rs"
server_key: "ae380e24b9fad5ed4053beedd74fb362"
server_key_iv: "5a8d06244b29aebbeb593115b0cb9ec5"
server_key_version: "encryptionkey@0"
I20260927 00:51:24.147141 21435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:24.169938 21435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:24.173511 21435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:24.173733 21435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:24.181445 21435 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46201
I20260927 00:51:24.181464 21502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46201 every 8 connection(s)
I20260927 00:51:24.182476 21435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:51:24.185191 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 21435
I20260927 00:51:24.185353 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:51:24.185686 392 external_mini_cluster.cc:1428] Setting key 8412240e93d0ffc76a7994c7fd659948
I20260927 00:51:24.185782 21503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:24.193113 21503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 946e02278c8e42e1967f5c0e440c44a6: Bootstrap starting.
I20260927 00:51:24.195590 21506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:24.187247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:43977 (local address 127.0.0.1:46201)
0927 00:51:24.187768 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:24.187784 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:24.187823 (+ 39us) server_negotiation.cc:407] Connection header received
0927 00:51:24.188816 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:24.188837 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:24.189224 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:24.189593 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:24.190026 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:24.191050 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:24.192017 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:24.192612 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:24.192809 (+ 197us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:24.193834 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:24.193839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:24.193852 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:24.194263 (+ 411us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:24.194322 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:24.194344 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:24.194491 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:24.194672 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 00:51:24.194837 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":172,"threads_started":1}
I20260927 00:51:24.195832 21503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 946e02278c8e42e1967f5c0e440c44a6: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:24.196738 21503 log.cc:824] T 00000000000000000000000000000000 P 946e02278c8e42e1967f5c0e440c44a6: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:24.198884 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:51:24.199051 21503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 946e02278c8e42e1967f5c0e440c44a6: No bootstrap required, opened a new log
I20260927 00:51:24.204484 21503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 946e02278c8e42e1967f5c0e440c44a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946e02278c8e42e1967f5c0e440c44a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46201 } }
I20260927 00:51:24.204870 21503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 946e02278c8e42e1967f5c0e440c44a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:24.204954 21503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 946e02278c8e42e1967f5c0e440c44a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 946e02278c8e42e1967f5c0e440c44a6, State: Initialized, Role: FOLLOWER
I20260927 00:51:24.205518 21503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 946e02278c8e42e1967f5c0e440c44a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946e02278c8e42e1967f5c0e440c44a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46201 } }
I20260927 00:51:24.205655 21503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 946e02278c8e42e1967f5c0e440c44a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:51:24.205749 21503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 946e02278c8e42e1967f5c0e440c44a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:51:24.205875 21503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 946e02278c8e42e1967f5c0e440c44a6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:24.206388 21506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:24.201492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:50813 (local address 127.0.0.1:46201)
0927 00:51:24.201646 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:24.201653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:24.201753 (+ 100us) server_negotiation.cc:407] Connection header received
0927 00:51:24.201877 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:24.201880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:24.201949 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:24.202074 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:24.202336 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:24.203202 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:24.204200 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:24.204617 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:24.204731 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:24.205860 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:24.205875 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:24.205878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:24.205897 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:24.205931 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:24.205951 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:24.206062 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:24.206193 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 00:51:24.206237 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:51:24.207247 21503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 946e02278c8e42e1967f5c0e440c44a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946e02278c8e42e1967f5c0e440c44a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46201 } }
I20260927 00:51:24.207681 21503 leader_election.cc:302] T 00000000000000000000000000000000 P 946e02278c8e42e1967f5c0e440c44a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 946e02278c8e42e1967f5c0e440c44a6; no voters:
I20260927 00:51:24.208042 21503 leader_election.cc:288] T 00000000000000000000000000000000 P 946e02278c8e42e1967f5c0e440c44a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:51:24.208166 21513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 946e02278c8e42e1967f5c0e440c44a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:24.208448 21513 raft_consensus.cc:686] T 00000000000000000000000000000000 P 946e02278c8e42e1967f5c0e440c44a6 [term 1 LEADER]: Becoming Leader. State: Replica: 946e02278c8e42e1967f5c0e440c44a6, State: Running, Role: LEADER
I20260927 00:51:24.208766 21513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 946e02278c8e42e1967f5c0e440c44a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946e02278c8e42e1967f5c0e440c44a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46201 } }
I20260927 00:51:24.209503 21503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 946e02278c8e42e1967f5c0e440c44a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:51:24.210487 21515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 946e02278c8e42e1967f5c0e440c44a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 946e02278c8e42e1967f5c0e440c44a6. Latest consensus state: current_term: 1 leader_uuid: "946e02278c8e42e1967f5c0e440c44a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946e02278c8e42e1967f5c0e440c44a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46201 } } }
I20260927 00:51:24.210624 21515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 946e02278c8e42e1967f5c0e440c44a6 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:24.210947 21514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 946e02278c8e42e1967f5c0e440c44a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "946e02278c8e42e1967f5c0e440c44a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946e02278c8e42e1967f5c0e440c44a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46201 } } }
I20260927 00:51:24.211042 21514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 946e02278c8e42e1967f5c0e440c44a6 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:24.211339 21522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:51:24.214386 21522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:51:24.221452 21522 catalog_manager.cc:1347] Generated new cluster ID: 9d09e0f9c39b4fa996128c50546a066a
I20260927 00:51:24.221549 21522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:51:24.237449 21522 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:51:24.238755 21522 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:51:24.251514 21522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 946e02278c8e42e1967f5c0e440c44a6: Generated new TSK 0
I20260927 00:51:24.252368 21522 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:51:24.265844 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 21435
W20260927 00:51:24.271871 21420 connection.cc:535] client connection to 127.0.0.1:46201 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46201): Connection reset by peer (error 104)
2026-09-27T00:51:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1451 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:51:24 dist-test-slave-86rs krb5kdc[21534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:51:24 dist-test-slave-86rs krb5kdc[21534](info): set up 2 sockets
Sep 27 00:51:24 dist-test-slave-86rs krb5kdc[21534](info): commencing operation
krb5kdc: starting...
W20260927 00:51:25.324736 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:51:25 dist-test-slave-86rs krb5kdc[21534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:51:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:51:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:51:25.455240 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37745
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:25.569866 21547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:25.570168 21547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:25.570211 21547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:25.574440 21547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:25.574522 21547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:25.574542 21547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:25.574559 21547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:25.574630 21547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:25.580325 21547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:25.581574 21547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:25.582711 21547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:25.591799 21552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:25.591801 21553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:25.591776 21555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:25.592056 21547 server_base.cc:1034] running on GCE node
I20260927 00:51:25.592855 21547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:25.594081 21547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:25.595289 21547 hybrid_clock.cc:648] HybridClock initialized: now 1790470285595253 us; error 54 us; skew 500 ppm
Sep 27 00:51:25 dist-test-slave-86rs krb5kdc[21534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:25.598564 21547 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:51:25.599596 21547 webserver.cc:460] Webserver started at http://127.0.0.1:40857/ using document root <none> and password file <none>
I20260927 00:51:25.600203 21547 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:25.600359 21547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:25.600623 21547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:25.602902 21547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "95329dcef4a1485eaaa3cd49223d6bbc"
format_stamp: "Formatted at 2026-09-27 00:51:25 on dist-test-slave-86rs"
server_key: "12bb87f8592d88452e573502a0553b46"
server_key_iv: "60c4b8bdece9e1629586a585582d74a3"
server_key_version: "encryptionkey@0"
I20260927 00:51:25.603513 21547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "95329dcef4a1485eaaa3cd49223d6bbc"
format_stamp: "Formatted at 2026-09-27 00:51:25 on dist-test-slave-86rs"
server_key: "12bb87f8592d88452e573502a0553b46"
server_key_iv: "60c4b8bdece9e1629586a585582d74a3"
server_key_version: "encryptionkey@0"
I20260927 00:51:25.607452 21547 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:51:25.610031 21562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:25.611371 21547 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:51:25.611531 21547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "95329dcef4a1485eaaa3cd49223d6bbc"
format_stamp: "Formatted at 2026-09-27 00:51:25 on dist-test-slave-86rs"
server_key: "12bb87f8592d88452e573502a0553b46"
server_key_iv: "60c4b8bdece9e1629586a585582d74a3"
server_key_version: "encryptionkey@0"
I20260927 00:51:25.611661 21547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:25.616464 21547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:25.619686 21547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:25.619910 21547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:25.627511 21547 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37745
I20260927 00:51:25.627532 21614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37745 every 8 connection(s)
I20260927 00:51:25.628597 21547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:51:25.631503 21615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:25.631826 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 21547
I20260927 00:51:25.631956 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:51:25.632335 392 external_mini_cluster.cc:1428] Setting key 3891add27307a26f047d1f288a7f116c
I20260927 00:51:25.638036 21618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:25.633915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:47439 (local address 127.0.0.1:37745)
0927 00:51:25.634376 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:25.634403 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:25.634442 (+ 39us) server_negotiation.cc:407] Connection header received
0927 00:51:25.635098 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:25.635115 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:25.635415 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:25.635704 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:25.636684 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:25.636689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:25.636701 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:25.636712 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:25.636776 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:25.636796 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:25.636909 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:25.637070 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 00:51:25.637276 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":163,"threads_started":1}
I20260927 00:51:25.638301 21615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95329dcef4a1485eaaa3cd49223d6bbc: Bootstrap starting.
I20260927 00:51:25.641026 21615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95329dcef4a1485eaaa3cd49223d6bbc: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:25.641072 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:51:25.641950 21615 log.cc:824] T 00000000000000000000000000000000 P 95329dcef4a1485eaaa3cd49223d6bbc: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:25.644726 21615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95329dcef4a1485eaaa3cd49223d6bbc: No bootstrap required, opened a new log
I20260927 00:51:25.645696 21618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:25.643639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:38757 (local address 127.0.0.1:37745)
0927 00:51:25.643787 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:25.643792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:25.644051 (+ 259us) server_negotiation.cc:407] Connection header received
0927 00:51:25.644149 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:25.644152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:25.644195 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:25.644384 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:25.645246 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:25.645249 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:25.645251 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:25.645261 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:25.645289 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:25.645300 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:25.645414 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:25.645487 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 00:51:25.645526 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:51:25.649222 21615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95329dcef4a1485eaaa3cd49223d6bbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95329dcef4a1485eaaa3cd49223d6bbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37745 } }
I20260927 00:51:25.649526 21615 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95329dcef4a1485eaaa3cd49223d6bbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:25.649590 21615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95329dcef4a1485eaaa3cd49223d6bbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95329dcef4a1485eaaa3cd49223d6bbc, State: Initialized, Role: FOLLOWER
I20260927 00:51:25.650015 21615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95329dcef4a1485eaaa3cd49223d6bbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95329dcef4a1485eaaa3cd49223d6bbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37745 } }
I20260927 00:51:25.650171 21615 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95329dcef4a1485eaaa3cd49223d6bbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:51:25.650264 21615 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95329dcef4a1485eaaa3cd49223d6bbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:51:25.650367 21615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95329dcef4a1485eaaa3cd49223d6bbc [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:25.651449 21615 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95329dcef4a1485eaaa3cd49223d6bbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95329dcef4a1485eaaa3cd49223d6bbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37745 } }
I20260927 00:51:25.651827 21615 leader_election.cc:302] T 00000000000000000000000000000000 P 95329dcef4a1485eaaa3cd49223d6bbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95329dcef4a1485eaaa3cd49223d6bbc; no voters:
I20260927 00:51:25.652204 21615 leader_election.cc:288] T 00000000000000000000000000000000 P 95329dcef4a1485eaaa3cd49223d6bbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:51:25.652380 21625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95329dcef4a1485eaaa3cd49223d6bbc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:25.652688 21625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95329dcef4a1485eaaa3cd49223d6bbc [term 1 LEADER]: Becoming Leader. State: Replica: 95329dcef4a1485eaaa3cd49223d6bbc, State: Running, Role: LEADER
I20260927 00:51:25.653071 21625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95329dcef4a1485eaaa3cd49223d6bbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95329dcef4a1485eaaa3cd49223d6bbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37745 } }
I20260927 00:51:25.653671 21615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95329dcef4a1485eaaa3cd49223d6bbc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:51:25.654870 21627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95329dcef4a1485eaaa3cd49223d6bbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95329dcef4a1485eaaa3cd49223d6bbc. Latest consensus state: current_term: 1 leader_uuid: "95329dcef4a1485eaaa3cd49223d6bbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95329dcef4a1485eaaa3cd49223d6bbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37745 } } }
I20260927 00:51:25.655002 21627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95329dcef4a1485eaaa3cd49223d6bbc [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:25.655303 21626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95329dcef4a1485eaaa3cd49223d6bbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95329dcef4a1485eaaa3cd49223d6bbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95329dcef4a1485eaaa3cd49223d6bbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37745 } } }
I20260927 00:51:25.655382 21626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95329dcef4a1485eaaa3cd49223d6bbc [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:25.655817 21634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:51:25.658819 21634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:51:25.665390 21634 catalog_manager.cc:1347] Generated new cluster ID: 229c3f558a5a40978de3678d985d0621
I20260927 00:51:25.665482 21634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:51:25.691861 21634 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:51:25.692279 21634 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:51:25.701707 21634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95329dcef4a1485eaaa3cd49223d6bbc: Generated new TSK 0
I20260927 00:51:25.702663 21634 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:51:25.710938 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 21547
2026-09-27T00:51:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1444 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:51:25 dist-test-slave-86rs krb5kdc[21646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:51:25 dist-test-slave-86rs krb5kdc[21646](info): set up 2 sockets
Sep 27 00:51:25 dist-test-slave-86rs krb5kdc[21646](info): commencing operation
krb5kdc: starting...
W20260927 00:51:26.756702 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:51:26 dist-test-slave-86rs krb5kdc[21646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:51:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:51:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:51:26.885473 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44887
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:27.000950 21659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:27.001374 21659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:27.001426 21659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:27.005613 21659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:27.005687 21659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:27.005708 21659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:27.005725 21659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:27.005740 21659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:27.011560 21659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44887
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:27.012751 21659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:27.013778 21659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:27.022998 21667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:27.022944 21665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:27.022974 21664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:27.023154 21659 server_base.cc:1034] running on GCE node
I20260927 00:51:27.023895 21659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:27.025234 21659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:27.026439 21659 hybrid_clock.cc:648] HybridClock initialized: now 1790470287026420 us; error 38 us; skew 500 ppm
Sep 27 00:51:27 dist-test-slave-86rs krb5kdc[21646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:27.029433 21659 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:51:27.030419 21659 webserver.cc:460] Webserver started at http://127.0.0.1:41207/ using document root <none> and password file <none>
I20260927 00:51:27.031041 21659 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:27.031139 21659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:27.031388 21659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:27.033632 21659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "78384e12c87e45c49428838f39503ad3"
format_stamp: "Formatted at 2026-09-27 00:51:27 on dist-test-slave-86rs"
server_key: "c997b8768c6e1fd80e9196b8b1dcc407"
server_key_iv: "b3fc8aabc5500ff9c58be22d391895e2"
server_key_version: "encryptionkey@0"
I20260927 00:51:27.034247 21659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "78384e12c87e45c49428838f39503ad3"
format_stamp: "Formatted at 2026-09-27 00:51:27 on dist-test-slave-86rs"
server_key: "c997b8768c6e1fd80e9196b8b1dcc407"
server_key_iv: "b3fc8aabc5500ff9c58be22d391895e2"
server_key_version: "encryptionkey@0"
I20260927 00:51:27.038277 21659 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:51:27.041136 21674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:27.042615 21659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:51:27.042814 21659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "78384e12c87e45c49428838f39503ad3"
format_stamp: "Formatted at 2026-09-27 00:51:27 on dist-test-slave-86rs"
server_key: "c997b8768c6e1fd80e9196b8b1dcc407"
server_key_iv: "b3fc8aabc5500ff9c58be22d391895e2"
server_key_version: "encryptionkey@0"
I20260927 00:51:27.042964 21659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:27.081009 21659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:27.084427 21659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:27.084642 21659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:27.092697 21659 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44887
I20260927 00:51:27.092713 21726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44887 every 8 connection(s)
I20260927 00:51:27.093775 21659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:51:27.096786 21727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:27.102852 21727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78384e12c87e45c49428838f39503ad3: Bootstrap starting.
I20260927 00:51:27.103125 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 21659
I20260927 00:51:27.103284 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:51:27.103582 392 external_mini_cluster.cc:1428] Setting key e3bd925ca64435f224bbbc929bf6ee2d
I20260927 00:51:27.106055 21727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78384e12c87e45c49428838f39503ad3: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:27.106986 21727 log.cc:824] T 00000000000000000000000000000000 P 78384e12c87e45c49428838f39503ad3: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:27.109439 21727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78384e12c87e45c49428838f39503ad3: No bootstrap required, opened a new log
Sep 27 00:51:27 dist-test-slave-86rs krb5kdc[21646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 00:51:27.113927 21727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78384e12c87e45c49428838f39503ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78384e12c87e45c49428838f39503ad3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44887 } }
I20260927 00:51:27.114248 21727 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78384e12c87e45c49428838f39503ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:27.114333 21727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78384e12c87e45c49428838f39503ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78384e12c87e45c49428838f39503ad3, State: Initialized, Role: FOLLOWER
I20260927 00:51:27.114904 21727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78384e12c87e45c49428838f39503ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78384e12c87e45c49428838f39503ad3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44887 } }
I20260927 00:51:27.115100 21727 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78384e12c87e45c49428838f39503ad3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:51:27.115172 21727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78384e12c87e45c49428838f39503ad3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:51:27.115309 21727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78384e12c87e45c49428838f39503ad3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:27.116356 21727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78384e12c87e45c49428838f39503ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78384e12c87e45c49428838f39503ad3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44887 } }
I20260927 00:51:27.116727 21727 leader_election.cc:302] T 00000000000000000000000000000000 P 78384e12c87e45c49428838f39503ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78384e12c87e45c49428838f39503ad3; no voters:
I20260927 00:51:27.117100 21727 leader_election.cc:288] T 00000000000000000000000000000000 P 78384e12c87e45c49428838f39503ad3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:51:27.117219 21732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78384e12c87e45c49428838f39503ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:27.117471 21732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78384e12c87e45c49428838f39503ad3 [term 1 LEADER]: Becoming Leader. State: Replica: 78384e12c87e45c49428838f39503ad3, State: Running, Role: LEADER
I20260927 00:51:27.117823 21732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78384e12c87e45c49428838f39503ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78384e12c87e45c49428838f39503ad3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44887 } }
I20260927 00:51:27.118620 21727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78384e12c87e45c49428838f39503ad3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:51:27.119158 21732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78384e12c87e45c49428838f39503ad3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78384e12c87e45c49428838f39503ad3. Latest consensus state: current_term: 1 leader_uuid: "78384e12c87e45c49428838f39503ad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78384e12c87e45c49428838f39503ad3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44887 } } }
I20260927 00:51:27.119330 21732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78384e12c87e45c49428838f39503ad3 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:27.120172 21737 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:51:27.120864 21730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:27.105186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:51257 (local address 127.0.0.1:44887)
0927 00:51:27.105642 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:27.105662 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:27.105716 (+ 54us) server_negotiation.cc:407] Connection header received
0927 00:51:27.106645 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:27.106671 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:27.107155 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:27.107536 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:27.108420 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:27.109900 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:27.110789 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:27.111521 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:27.111696 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:27.114680 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:27.114686 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:27.114704 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:27.114752 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:27.116932 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:27.117621 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:27.117628 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:27.117634 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:27.117817 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:27.118245 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:27.118249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:27.118251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:27.118821 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:27.119052 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:27.119449 (+ 397us) server_negotiation.cc:299] Negotiation successful
0927 00:51:27.119701 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":131,"threads_started":1}
I20260927 00:51:27.119949 21733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78384e12c87e45c49428838f39503ad3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78384e12c87e45c49428838f39503ad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78384e12c87e45c49428838f39503ad3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44887 } } }
I20260927 00:51:27.121233 21733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78384e12c87e45c49428838f39503ad3 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:27.124063 21737 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:51:27.124892 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:51:27.130643 21737 catalog_manager.cc:1347] Generated new cluster ID: 3936f54fa530442d850a595fa1bb7d99
I20260927 00:51:27.130795 21737 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:51:27.136844 21730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:27.127223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:39909 (local address 127.0.0.1:44887)
0927 00:51:27.127359 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:27.127363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:27.127574 (+ 211us) server_negotiation.cc:407] Connection header received
0927 00:51:27.127703 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:27.127707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:27.127762 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:27.127854 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:27.128776 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:27.129659 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:27.130898 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:27.131230 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:27.131344 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:27.133190 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:27.133194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:27.133197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:27.133226 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:27.134854 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:27.135457 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:27.135460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:27.135462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:27.135542 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:27.135907 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:27.135910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:27.135911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:27.136107 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:27.136288 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:27.136577 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 00:51:27.136675 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:51:27.165441 21737 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:51:27.166834 21737 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:51:27.174587 21737 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78384e12c87e45c49428838f39503ad3: Generated new TSK 0
I20260927 00:51:27.175568 21737 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:51:27.182057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 21659
W20260927 00:51:27.187783 21644 connection.cc:535] client connection to 127.0.0.1:44887 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44887): Connection reset by peer (error 104)
2026-09-27T00:51:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1471 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:51:27 dist-test-slave-86rs krb5kdc[21758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:51:27 dist-test-slave-86rs krb5kdc[21758](info): set up 2 sockets
Sep 27 00:51:27 dist-test-slave-86rs krb5kdc[21758](info): commencing operation
krb5kdc: starting...
W20260927 00:51:28.220554 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:51:28 dist-test-slave-86rs krb5kdc[21758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:51:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:51:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:51:28.347586 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39711
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:28.464318 21771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:28.464694 21771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:28.464797 21771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:28.469095 21771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:28.469202 21771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:28.469238 21771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:28.469280 21771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:28.469319 21771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:28.474895 21771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39711
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:28.476089 21771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:28.477255 21771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:28.486091 21776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:28.486347 21771 server_base.cc:1034] running on GCE node
W20260927 00:51:28.486079 21779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:28.486079 21777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:28.487071 21771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:28.488373 21771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:28.489569 21771 hybrid_clock.cc:648] HybridClock initialized: now 1790470288489557 us; error 33 us; skew 500 ppm
Sep 27 00:51:28 dist-test-slave-86rs krb5kdc[21758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:28.492586 21771 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:51:28.493584 21771 webserver.cc:460] Webserver started at http://127.0.0.1:42295/ using document root <none> and password file <none>
I20260927 00:51:28.494230 21771 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:28.494333 21771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:28.494580 21771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:28.496902 21771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "59f46baf51194f4bbff5b90d975435a1"
format_stamp: "Formatted at 2026-09-27 00:51:28 on dist-test-slave-86rs"
server_key: "4fdaf7c03653d5a82767015f143e515d"
server_key_iv: "1ae2acf5ca8f6e12ea7164a8c1529310"
server_key_version: "encryptionkey@0"
I20260927 00:51:28.497524 21771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "59f46baf51194f4bbff5b90d975435a1"
format_stamp: "Formatted at 2026-09-27 00:51:28 on dist-test-slave-86rs"
server_key: "4fdaf7c03653d5a82767015f143e515d"
server_key_iv: "1ae2acf5ca8f6e12ea7164a8c1529310"
server_key_version: "encryptionkey@0"
I20260927 00:51:28.501561 21771 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:51:28.504191 21786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:28.505566 21771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:51:28.505723 21771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "59f46baf51194f4bbff5b90d975435a1"
format_stamp: "Formatted at 2026-09-27 00:51:28 on dist-test-slave-86rs"
server_key: "4fdaf7c03653d5a82767015f143e515d"
server_key_iv: "1ae2acf5ca8f6e12ea7164a8c1529310"
server_key_version: "encryptionkey@0"
I20260927 00:51:28.505861 21771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:28.524158 21771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:28.527829 21771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:28.528057 21771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:28.536362 21771 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39711
I20260927 00:51:28.536362 21838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39711 every 8 connection(s)
I20260927 00:51:28.537426 21771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:51:28.540333 21839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:28.544833 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 21771
I20260927 00:51:28.544988 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:51:28.545252 392 external_mini_cluster.cc:1428] Setting key 65f0ddea1c79ff820d4d2b753e147b77
I20260927 00:51:28.546527 21839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59f46baf51194f4bbff5b90d975435a1: Bootstrap starting.
I20260927 00:51:28.549032 21839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59f46baf51194f4bbff5b90d975435a1: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:28.549803 21839 log.cc:824] T 00000000000000000000000000000000 P 59f46baf51194f4bbff5b90d975435a1: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:28.551770 21839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59f46baf51194f4bbff5b90d975435a1: No bootstrap required, opened a new log
I20260927 00:51:28.554594 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:28.546756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:49479 (local address 127.0.0.1:39711)
0927 00:51:28.547220 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:28.547235 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:28.547270 (+ 35us) server_negotiation.cc:407] Connection header received
0927 00:51:28.547968 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:28.547986 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:28.548438 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:28.548794 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:28.549273 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:28.550333 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:28.551262 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:28.551743 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:28.551923 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:28.552879 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:28.552885 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:28.552898 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:28.553296 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:28.553353 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:28.553375 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:28.553524 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:28.553702 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 00:51:28.553845 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":145,"threads_started":1}
I20260927 00:51:28.555646 21839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59f46baf51194f4bbff5b90d975435a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59f46baf51194f4bbff5b90d975435a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39711 } }
I20260927 00:51:28.555912 21839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59f46baf51194f4bbff5b90d975435a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:28.556005 21839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59f46baf51194f4bbff5b90d975435a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f46baf51194f4bbff5b90d975435a1, State: Initialized, Role: FOLLOWER
I20260927 00:51:28.556522 21839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59f46baf51194f4bbff5b90d975435a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59f46baf51194f4bbff5b90d975435a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39711 } }
I20260927 00:51:28.556710 21839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59f46baf51194f4bbff5b90d975435a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:51:28.556780 21839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59f46baf51194f4bbff5b90d975435a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:51:28.556879 21839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59f46baf51194f4bbff5b90d975435a1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:28.557812 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:51:28.558135 21839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59f46baf51194f4bbff5b90d975435a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59f46baf51194f4bbff5b90d975435a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39711 } }
I20260927 00:51:28.558555 21839 leader_election.cc:302] T 00000000000000000000000000000000 P 59f46baf51194f4bbff5b90d975435a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59f46baf51194f4bbff5b90d975435a1; no voters:
I20260927 00:51:28.558924 21839 leader_election.cc:288] T 00000000000000000000000000000000 P 59f46baf51194f4bbff5b90d975435a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:51:28.559028 21844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59f46baf51194f4bbff5b90d975435a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:28.559264 21844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59f46baf51194f4bbff5b90d975435a1 [term 1 LEADER]: Becoming Leader. State: Replica: 59f46baf51194f4bbff5b90d975435a1, State: Running, Role: LEADER
I20260927 00:51:28.559640 21844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59f46baf51194f4bbff5b90d975435a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59f46baf51194f4bbff5b90d975435a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39711 } }
I20260927 00:51:28.560367 21839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59f46baf51194f4bbff5b90d975435a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:51:28.561437 21847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59f46baf51194f4bbff5b90d975435a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59f46baf51194f4bbff5b90d975435a1. Latest consensus state: current_term: 1 leader_uuid: "59f46baf51194f4bbff5b90d975435a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59f46baf51194f4bbff5b90d975435a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39711 } } }
I20260927 00:51:28.561558 21847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59f46baf51194f4bbff5b90d975435a1 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:28.561895 21845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59f46baf51194f4bbff5b90d975435a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59f46baf51194f4bbff5b90d975435a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59f46baf51194f4bbff5b90d975435a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39711 } } }
I20260927 00:51:28.561978 21845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59f46baf51194f4bbff5b90d975435a1 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:28.562819 21858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:51:28.565547 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:28.560128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:55817 (local address 127.0.0.1:39711)
0927 00:51:28.560279 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:28.560293 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:28.560865 (+ 572us) server_negotiation.cc:407] Connection header received
0927 00:51:28.561031 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:28.561035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:28.561087 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:28.561146 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:28.561630 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:28.562484 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:28.563623 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:28.563948 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:28.564066 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:28.565030 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:28.565034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:28.565037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:28.565051 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:28.565082 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:28.565096 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:28.565188 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:28.565372 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 00:51:28.565410 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:51:28.566201 21858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:51:28.571718 21858 catalog_manager.cc:1347] Generated new cluster ID: 3d345895070d4ccf93a2d4889a3e7d32
I20260927 00:51:28.571801 21858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:51:28.587280 21858 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:51:28.588407 21858 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:51:28.593456 21858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59f46baf51194f4bbff5b90d975435a1: Generated new TSK 0
I20260927 00:51:28.594246 21858 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:51:28.599097 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 21771
W20260927 00:51:28.604488 21756 connection.cc:535] client connection to 127.0.0.1:39711 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39711): Connection reset by peer (error 104)
2026-09-27T00:51:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:51:28 dist-test-slave-86rs krb5kdc[21870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:51:28 dist-test-slave-86rs krb5kdc[21870](info): set up 2 sockets
Sep 27 00:51:28 dist-test-slave-86rs krb5kdc[21870](info): commencing operation
krb5kdc: starting...
W20260927 00:51:29.636610 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:51:29 dist-test-slave-86rs krb5kdc[21870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:51:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:51:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:51:29.764050 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43151
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:29.878281 21883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:29.878598 21883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:29.878645 21883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:29.882992 21883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:29.883076 21883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:29.883096 21883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:29.883114 21883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:29.883129 21883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:29.888986 21883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43151
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470006085796-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:29.890187 21883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:29.891299 21883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:29.900187 21891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:29.900189 21888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:29.900187 21889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:29.900297 21883 server_base.cc:1034] running on GCE node
I20260927 00:51:29.901028 21883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:29.902223 21883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:29.903419 21883 hybrid_clock.cc:648] HybridClock initialized: now 1790470289903387 us; error 50 us; skew 500 ppm
Sep 27 00:51:29 dist-test-slave-86rs krb5kdc[21870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:29.910403 21883 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:51:29.911490 21883 webserver.cc:460] Webserver started at http://127.0.0.1:41729/ using document root <none> and password file <none>
I20260927 00:51:29.912106 21883 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:29.912206 21883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:29.912467 21883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:29.914664 21883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "7d14a7019b14432e8f8e4e1ce4ea7c0e"
format_stamp: "Formatted at 2026-09-27 00:51:29 on dist-test-slave-86rs"
server_key: "9a3b41076e06f2170e6324bf1f09e2bc"
server_key_iv: "09bd4cdb37fada778266ccb5bb2c6c71"
server_key_version: "encryptionkey@0"
I20260927 00:51:29.915467 21883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "7d14a7019b14432e8f8e4e1ce4ea7c0e"
format_stamp: "Formatted at 2026-09-27 00:51:29 on dist-test-slave-86rs"
server_key: "9a3b41076e06f2170e6324bf1f09e2bc"
server_key_iv: "09bd4cdb37fada778266ccb5bb2c6c71"
server_key_version: "encryptionkey@0"
I20260927 00:51:29.919689 21883 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260927 00:51:29.922457 21898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:29.923708 21883 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 00:51:29.923868 21883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "7d14a7019b14432e8f8e4e1ce4ea7c0e"
format_stamp: "Formatted at 2026-09-27 00:51:29 on dist-test-slave-86rs"
server_key: "9a3b41076e06f2170e6324bf1f09e2bc"
server_key_iv: "09bd4cdb37fada778266ccb5bb2c6c71"
server_key_version: "encryptionkey@0"
I20260927 00:51:29.924008 21883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:29.928934 21883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:29.932399 21883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:29.932611 21883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:29.940332 21883 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43151
I20260927 00:51:29.940361 21950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43151 every 8 connection(s)
I20260927 00:51:29.941373 21883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:51:29.944432 21951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:29.950234 21951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d14a7019b14432e8f8e4e1ce4ea7c0e: Bootstrap starting.
I20260927 00:51:29.950834 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 21883
I20260927 00:51:29.950973 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:51:29.951267 392 external_mini_cluster.cc:1428] Setting key b0116b2d442cd83d24490e953523c896
I20260927 00:51:29.953279 21951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d14a7019b14432e8f8e4e1ce4ea7c0e: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:29.954044 21951 log.cc:824] T 00000000000000000000000000000000 P 7d14a7019b14432e8f8e4e1ce4ea7c0e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:29.956314 21951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d14a7019b14432e8f8e4e1ce4ea7c0e: No bootstrap required, opened a new log
I20260927 00:51:29.956957 21954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:29.952685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:60659 (local address 127.0.0.1:43151)
0927 00:51:29.953171 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:29.953187 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:29.953223 (+ 36us) server_negotiation.cc:407] Connection header received
0927 00:51:29.953990 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:29.954009 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:29.954360 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:29.954712 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:29.955565 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:29.955570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:29.955583 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:29.955593 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:29.955650 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:29.955669 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:29.955812 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:29.955932 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 00:51:29.956101 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":146,"threads_started":1}
I20260927 00:51:29.959805 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:51:29.960259 21951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d14a7019b14432e8f8e4e1ce4ea7c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d14a7019b14432e8f8e4e1ce4ea7c0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43151 } }
I20260927 00:51:29.960512 21951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d14a7019b14432e8f8e4e1ce4ea7c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:29.960593 21951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d14a7019b14432e8f8e4e1ce4ea7c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d14a7019b14432e8f8e4e1ce4ea7c0e, State: Initialized, Role: FOLLOWER
I20260927 00:51:29.961148 21951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d14a7019b14432e8f8e4e1ce4ea7c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d14a7019b14432e8f8e4e1ce4ea7c0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43151 } }
I20260927 00:51:29.961321 21951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d14a7019b14432e8f8e4e1ce4ea7c0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:51:29.961416 21951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d14a7019b14432e8f8e4e1ce4ea7c0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:51:29.961522 21951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d14a7019b14432e8f8e4e1ce4ea7c0e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:29.962975 21951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d14a7019b14432e8f8e4e1ce4ea7c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d14a7019b14432e8f8e4e1ce4ea7c0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43151 } }
I20260927 00:51:29.963366 21951 leader_election.cc:302] T 00000000000000000000000000000000 P 7d14a7019b14432e8f8e4e1ce4ea7c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d14a7019b14432e8f8e4e1ce4ea7c0e; no voters:
I20260927 00:51:29.963788 21951 leader_election.cc:288] T 00000000000000000000000000000000 P 7d14a7019b14432e8f8e4e1ce4ea7c0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:51:29.963869 21961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d14a7019b14432e8f8e4e1ce4ea7c0e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:29.964013 21954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:29.962150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:49371 (local address 127.0.0.1:43151)
0927 00:51:29.962307 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:29.962311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:29.962488 (+ 177us) server_negotiation.cc:407] Connection header received
0927 00:51:29.962624 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:29.962627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:29.962659 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:29.962725 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:29.963459 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:29.963462 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:29.963464 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:29.963474 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:29.963500 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:29.963541 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:29.963635 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:29.963769 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 00:51:29.963816 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 00:51:29.964155 21961 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d14a7019b14432e8f8e4e1ce4ea7c0e [term 1 LEADER]: Becoming Leader. State: Replica: 7d14a7019b14432e8f8e4e1ce4ea7c0e, State: Running, Role: LEADER
I20260927 00:51:29.964524 21961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d14a7019b14432e8f8e4e1ce4ea7c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d14a7019b14432e8f8e4e1ce4ea7c0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43151 } }
I20260927 00:51:29.965245 21951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d14a7019b14432e8f8e4e1ce4ea7c0e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:51:29.965747 21961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d14a7019b14432e8f8e4e1ce4ea7c0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d14a7019b14432e8f8e4e1ce4ea7c0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d14a7019b14432e8f8e4e1ce4ea7c0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43151 } } }
I20260927 00:51:29.965865 21961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d14a7019b14432e8f8e4e1ce4ea7c0e [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:29.966063 21961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d14a7019b14432e8f8e4e1ce4ea7c0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d14a7019b14432e8f8e4e1ce4ea7c0e. Latest consensus state: current_term: 1 leader_uuid: "7d14a7019b14432e8f8e4e1ce4ea7c0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d14a7019b14432e8f8e4e1ce4ea7c0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43151 } } }
I20260927 00:51:29.966205 21961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d14a7019b14432e8f8e4e1ce4ea7c0e [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:29.966629 21966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:51:29.969408 21966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:51:29.975611 21966 catalog_manager.cc:1347] Generated new cluster ID: 937387d6a2564d768262870aef9219cc
I20260927 00:51:29.975710 21966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:51:29.990552 21966 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:51:29.990913 21966 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:51:29.996268 21966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d14a7019b14432e8f8e4e1ce4ea7c0e: Generated new TSK 0
I20260927 00:51:29.997033 21966 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:51:30.005481 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 21883
2026-09-27T00:51:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:51:30 dist-test-slave-86rs krb5kdc[21982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:51:30 dist-test-slave-86rs krb5kdc[21982](info): set up 2 sockets
Sep 27 00:51:30 dist-test-slave-86rs krb5kdc[21982](info): commencing operation
krb5kdc: starting...
W20260927 00:51:31.056535 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:51:31 dist-test-slave-86rs krb5kdc[21982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:51:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:51:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:51:31.189037 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44923
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:31.304076 21995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:31.304495 21995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:31.304613 21995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:31.308857 21995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:31.308970 21995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:31.309010 21995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:31.309063 21995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:31.309098 21995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:31.314855 21995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44923
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:31.316114 21995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:31.317262 21995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:31.326336 22001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:31.326385 22000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:31.326303 22003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:31.327039 21995 server_base.cc:1034] running on GCE node
I20260927 00:51:31.327636 21995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:31.328940 21995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:31.330137 21995 hybrid_clock.cc:648] HybridClock initialized: now 1790470291330097 us; error 51 us; skew 500 ppm
Sep 27 00:51:31 dist-test-slave-86rs krb5kdc[21982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:31.333125 21995 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:51:31.334187 21995 webserver.cc:460] Webserver started at http://127.0.98.62:36285/ using document root <none> and password file <none>
I20260927 00:51:31.334807 21995 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:31.334908 21995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:31.335158 21995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:31.337499 21995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "543bb5e8570e4cc69e8cb0b69c363c3b"
format_stamp: "Formatted at 2026-09-27 00:51:31 on dist-test-slave-86rs"
server_key: "0685174c8949a7b379cd6f75c7543e51"
server_key_iv: "c8879fe31e359c66dd680498d054d3b0"
server_key_version: "encryptionkey@0"
I20260927 00:51:31.338091 21995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "543bb5e8570e4cc69e8cb0b69c363c3b"
format_stamp: "Formatted at 2026-09-27 00:51:31 on dist-test-slave-86rs"
server_key: "0685174c8949a7b379cd6f75c7543e51"
server_key_iv: "c8879fe31e359c66dd680498d054d3b0"
server_key_version: "encryptionkey@0"
I20260927 00:51:31.342039 21995 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:51:31.344836 22010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:31.346259 21995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:51:31.346426 21995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "543bb5e8570e4cc69e8cb0b69c363c3b"
format_stamp: "Formatted at 2026-09-27 00:51:31 on dist-test-slave-86rs"
server_key: "0685174c8949a7b379cd6f75c7543e51"
server_key_iv: "c8879fe31e359c66dd680498d054d3b0"
server_key_version: "encryptionkey@0"
I20260927 00:51:31.346570 21995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:31.367355 21995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:31.370463 21995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:31.370683 21995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:31.378545 21995 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44923
I20260927 00:51:31.378579 22062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44923 every 8 connection(s)
I20260927 00:51:31.379581 21995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:51:31.382671 22063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:31.386409 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 21995
I20260927 00:51:31.386580 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:51:31.386873 392 external_mini_cluster.cc:1428] Setting key 2caf3d66a3638d9953e7455fed7e147b
I20260927 00:51:31.389817 22063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 543bb5e8570e4cc69e8cb0b69c363c3b: Bootstrap starting.
I20260927 00:51:31.392993 22063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 543bb5e8570e4cc69e8cb0b69c363c3b: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:31.393939 22063 log.cc:824] T 00000000000000000000000000000000 P 543bb5e8570e4cc69e8cb0b69c363c3b: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:31.396760 22063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 543bb5e8570e4cc69e8cb0b69c363c3b: No bootstrap required, opened a new log
Sep 27 00:51:31 dist-test-slave-86rs krb5kdc[21982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:51:31.401410 22063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 543bb5e8570e4cc69e8cb0b69c363c3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543bb5e8570e4cc69e8cb0b69c363c3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44923 } }
I20260927 00:51:31.401742 22063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 543bb5e8570e4cc69e8cb0b69c363c3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:31.401820 22063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 543bb5e8570e4cc69e8cb0b69c363c3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 543bb5e8570e4cc69e8cb0b69c363c3b, State: Initialized, Role: FOLLOWER
I20260927 00:51:31.402350 22063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 543bb5e8570e4cc69e8cb0b69c363c3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543bb5e8570e4cc69e8cb0b69c363c3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44923 } }
I20260927 00:51:31.402529 22063 raft_consensus.cc:388] T 00000000000000000000000000000000 P 543bb5e8570e4cc69e8cb0b69c363c3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:51:31.402603 22063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 543bb5e8570e4cc69e8cb0b69c363c3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:51:31.402717 22063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 543bb5e8570e4cc69e8cb0b69c363c3b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:31.403759 22063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 543bb5e8570e4cc69e8cb0b69c363c3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543bb5e8570e4cc69e8cb0b69c363c3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44923 } }
I20260927 00:51:31.404142 22063 leader_election.cc:302] T 00000000000000000000000000000000 P 543bb5e8570e4cc69e8cb0b69c363c3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 543bb5e8570e4cc69e8cb0b69c363c3b; no voters:
I20260927 00:51:31.404577 22063 leader_election.cc:288] T 00000000000000000000000000000000 P 543bb5e8570e4cc69e8cb0b69c363c3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:51:31.404659 22068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 543bb5e8570e4cc69e8cb0b69c363c3b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:31.404884 22068 raft_consensus.cc:686] T 00000000000000000000000000000000 P 543bb5e8570e4cc69e8cb0b69c363c3b [term 1 LEADER]: Becoming Leader. State: Replica: 543bb5e8570e4cc69e8cb0b69c363c3b, State: Running, Role: LEADER
I20260927 00:51:31.405182 22066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:31.388514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34345 (local address 127.0.98.62:44923)
0927 00:51:31.388985 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:31.389006 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:31.389052 (+ 46us) server_negotiation.cc:407] Connection header received
0927 00:51:31.389974 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:31.389999 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:31.390483 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:31.390899 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:31.391637 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:31.393360 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:31.394428 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:31.395353 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:31.395540 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:31.398674 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:31.398681 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:31.398701 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:31.398741 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:31.401541 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:31.402258 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:31.402264 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:31.402269 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:31.402402 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:31.402828 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:31.402833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:31.402834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:31.403264 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:31.403521 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:31.403848 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 00:51:31.404113 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":120,"threads_started":1}
I20260927 00:51:31.405267 22068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 543bb5e8570e4cc69e8cb0b69c363c3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543bb5e8570e4cc69e8cb0b69c363c3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44923 } }
I20260927 00:51:31.405978 22063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 543bb5e8570e4cc69e8cb0b69c363c3b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:51:31.409693 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:51:31.408949 22070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 543bb5e8570e4cc69e8cb0b69c363c3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "543bb5e8570e4cc69e8cb0b69c363c3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543bb5e8570e4cc69e8cb0b69c363c3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44923 } } }
I20260927 00:51:31.409989 22070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 543bb5e8570e4cc69e8cb0b69c363c3b [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:31.409802 22069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 543bb5e8570e4cc69e8cb0b69c363c3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 543bb5e8570e4cc69e8cb0b69c363c3b. Latest consensus state: current_term: 1 leader_uuid: "543bb5e8570e4cc69e8cb0b69c363c3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543bb5e8570e4cc69e8cb0b69c363c3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44923 } } }
I20260927 00:51:31.410465 22069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 543bb5e8570e4cc69e8cb0b69c363c3b [sys.catalog]: This master's current role is: LEADER
W20260927 00:51:31.413311 22084 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 543bb5e8570e4cc69e8cb0b69c363c3b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:51:31.413414 22084 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:51:31.413477 22085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:51:31.414449 22085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:51:31.421123 22085 catalog_manager.cc:1347] Generated new cluster ID: 96427b3eb4d04230bff96b3e31fc8f45
I20260927 00:51:31.421192 22085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:51:31.421484 22066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:31.412128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40805 (local address 127.0.98.62:44923)
0927 00:51:31.412279 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:31.412284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:31.412733 (+ 449us) server_negotiation.cc:407] Connection header received
0927 00:51:31.412881 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:31.412884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:31.412935 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:31.413026 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:31.413834 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:31.414368 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:31.415243 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:31.415508 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:31.415620 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:31.417402 (+ 1782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:31.417407 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:31.417410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:31.417439 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:31.419239 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:31.419795 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:31.419799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:31.419801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:31.419891 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:31.420245 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:31.420249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:31.420254 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:31.420448 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:31.420606 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:31.420932 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 00:51:31.421042 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:51:31.429134 22085 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:51:31.430413 22085 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:51:31.435602 22085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 543bb5e8570e4cc69e8cb0b69c363c3b: Generated new TSK 0
I20260927 00:51:31.436417 22085 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:51:31.444060 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 21995
W20260927 00:51:31.449605 21980 connection.cc:535] client connection to 127.0.98.62:44923 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:44923): Connection reset by peer (error 104)
2026-09-27T00:51:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1438 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:51:31 dist-test-slave-86rs krb5kdc[22095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:51:31 dist-test-slave-86rs krb5kdc[22095](info): set up 2 sockets
Sep 27 00:51:31 dist-test-slave-86rs krb5kdc[22095](info): commencing operation
krb5kdc: starting...
W20260927 00:51:32.480599 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:51:32 dist-test-slave-86rs krb5kdc[22095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:51:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:51:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:51:32.603344 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36633
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:32.717988 22108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:32.718354 22108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:32.718442 22108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:32.722718 22108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:32.722834 22108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:32.722875 22108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:32.722936 22108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:32.722975 22108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:32.728708 22108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36633
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:32.729892 22108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:32.730988 22108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:32.739826 22113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:32.739876 22116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:32.739982 22108 server_base.cc:1034] running on GCE node
W20260927 00:51:32.739826 22114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:32.740768 22108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:32.742027 22108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:32.743238 22108 hybrid_clock.cc:648] HybridClock initialized: now 1790470292743222 us; error 30 us; skew 500 ppm
Sep 27 00:51:32 dist-test-slave-86rs krb5kdc[22095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:32.746223 22108 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:51:32.747203 22108 webserver.cc:460] Webserver started at http://127.0.98.62:44053/ using document root <none> and password file <none>
I20260927 00:51:32.747795 22108 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:32.747901 22108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:32.748143 22108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:32.750438 22108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "d23320cb54a64584ab67cc6991a46016"
format_stamp: "Formatted at 2026-09-27 00:51:32 on dist-test-slave-86rs"
server_key: "b2fce919639b6f59623481f48d5bb810"
server_key_iv: "f3fa072e30b366c319d0de14144999c8"
server_key_version: "encryptionkey@0"
I20260927 00:51:32.751053 22108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "d23320cb54a64584ab67cc6991a46016"
format_stamp: "Formatted at 2026-09-27 00:51:32 on dist-test-slave-86rs"
server_key: "b2fce919639b6f59623481f48d5bb810"
server_key_iv: "f3fa072e30b366c319d0de14144999c8"
server_key_version: "encryptionkey@0"
I20260927 00:51:32.755158 22108 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 00:51:32.757982 22123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:32.759310 22108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 00:51:32.759485 22108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "d23320cb54a64584ab67cc6991a46016"
format_stamp: "Formatted at 2026-09-27 00:51:32 on dist-test-slave-86rs"
server_key: "b2fce919639b6f59623481f48d5bb810"
server_key_iv: "f3fa072e30b366c319d0de14144999c8"
server_key_version: "encryptionkey@0"
I20260927 00:51:32.759637 22108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:32.783941 22108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:32.787271 22108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:32.787501 22108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:32.794798 22108 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36633
I20260927 00:51:32.794824 22175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36633 every 8 connection(s)
I20260927 00:51:32.795773 22108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:51:32.798889 22176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:32.800467 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 22108
I20260927 00:51:32.800624 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:51:32.800922 392 external_mini_cluster.cc:1428] Setting key 98d6c33349b14573481eabdea771923a
I20260927 00:51:32.806404 22176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d23320cb54a64584ab67cc6991a46016: Bootstrap starting.
I20260927 00:51:32.809238 22176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d23320cb54a64584ab67cc6991a46016: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:32.810185 22176 log.cc:824] T 00000000000000000000000000000000 P d23320cb54a64584ab67cc6991a46016: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:32.811583 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:32.802460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42905 (local address 127.0.98.62:36633)
0927 00:51:32.802928 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:32.802947 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:32.802990 (+ 43us) server_negotiation.cc:407] Connection header received
0927 00:51:32.803880 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:32.803904 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:32.804400 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:32.804769 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:32.805219 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:32.806696 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:32.807678 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:32.808399 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:32.808557 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:32.809540 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:32.809549 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:32.809570 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:32.810096 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:32.810163 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:32.810194 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:32.810336 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:32.810564 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 00:51:32.810714 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":126,"threads_started":1}
I20260927 00:51:32.812534 22176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d23320cb54a64584ab67cc6991a46016: No bootstrap required, opened a new log
I20260927 00:51:32.815027 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:51:32.817277 22176 raft_consensus.cc:348] T 00000000000000000000000000000000 P d23320cb54a64584ab67cc6991a46016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23320cb54a64584ab67cc6991a46016" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } }
I20260927 00:51:32.817618 22176 raft_consensus.cc:374] T 00000000000000000000000000000000 P d23320cb54a64584ab67cc6991a46016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:32.817706 22176 raft_consensus.cc:729] T 00000000000000000000000000000000 P d23320cb54a64584ab67cc6991a46016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d23320cb54a64584ab67cc6991a46016, State: Initialized, Role: FOLLOWER
I20260927 00:51:32.818229 22176 consensus_queue.cc:260] T 00000000000000000000000000000000 P d23320cb54a64584ab67cc6991a46016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23320cb54a64584ab67cc6991a46016" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } }
I20260927 00:51:32.818404 22176 raft_consensus.cc:388] T 00000000000000000000000000000000 P d23320cb54a64584ab67cc6991a46016 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:51:32.818480 22176 raft_consensus.cc:482] T 00000000000000000000000000000000 P d23320cb54a64584ab67cc6991a46016 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:51:32.818588 22176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d23320cb54a64584ab67cc6991a46016 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:32.819585 22176 raft_consensus.cc:504] T 00000000000000000000000000000000 P d23320cb54a64584ab67cc6991a46016 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23320cb54a64584ab67cc6991a46016" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } }
I20260927 00:51:32.819962 22176 leader_election.cc:302] T 00000000000000000000000000000000 P d23320cb54a64584ab67cc6991a46016 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d23320cb54a64584ab67cc6991a46016; no voters:
I20260927 00:51:32.820333 22176 leader_election.cc:288] T 00000000000000000000000000000000 P d23320cb54a64584ab67cc6991a46016 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:51:32.820408 22186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d23320cb54a64584ab67cc6991a46016 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:32.820611 22186 raft_consensus.cc:686] T 00000000000000000000000000000000 P d23320cb54a64584ab67cc6991a46016 [term 1 LEADER]: Becoming Leader. State: Replica: d23320cb54a64584ab67cc6991a46016, State: Running, Role: LEADER
I20260927 00:51:32.820914 22186 consensus_queue.cc:237] T 00000000000000000000000000000000 P d23320cb54a64584ab67cc6991a46016 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23320cb54a64584ab67cc6991a46016" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } }
I20260927 00:51:32.821727 22176 sys_catalog.cc:564] T 00000000000000000000000000000000 P d23320cb54a64584ab67cc6991a46016 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:51:32.822120 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:32.817178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39375 (local address 127.0.98.62:36633)
0927 00:51:32.817306 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:32.817309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:32.818004 (+ 695us) server_negotiation.cc:407] Connection header received
0927 00:51:32.818137 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:32.818140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:32.818191 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:32.818288 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:32.818634 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:32.819237 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:32.820078 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:32.820477 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:32.820580 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:32.821553 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:32.821556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:32.821559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:32.821572 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:32.821607 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:32.821619 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:32.821716 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:32.821923 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 00:51:32.821960 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:51:32.822108 22186 sys_catalog.cc:455] T 00000000000000000000000000000000 P d23320cb54a64584ab67cc6991a46016 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d23320cb54a64584ab67cc6991a46016. Latest consensus state: current_term: 1 leader_uuid: "d23320cb54a64584ab67cc6991a46016" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23320cb54a64584ab67cc6991a46016" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } } }
I20260927 00:51:32.822222 22186 sys_catalog.cc:458] T 00000000000000000000000000000000 P d23320cb54a64584ab67cc6991a46016 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:32.822431 22186 sys_catalog.cc:455] T 00000000000000000000000000000000 P d23320cb54a64584ab67cc6991a46016 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d23320cb54a64584ab67cc6991a46016" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23320cb54a64584ab67cc6991a46016" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } } }
I20260927 00:51:32.822525 22186 sys_catalog.cc:458] T 00000000000000000000000000000000 P d23320cb54a64584ab67cc6991a46016 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:32.823441 22190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:51:32.826403 22190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:51:32.832743 22190 catalog_manager.cc:1347] Generated new cluster ID: fa40253657374606a9f878ad3cdf80f3
I20260927 00:51:32.832841 22190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:51:32.843107 22190 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:51:32.844436 22190 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:51:32.849500 22190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d23320cb54a64584ab67cc6991a46016: Generated new TSK 0
I20260927 00:51:32.850193 22190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:51:32.857611 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 22108
W20260927 00:51:32.863118 22093 connection.cc:535] client connection to 127.0.98.62:36633 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:36633): Connection reset by peer (error 104)
2026-09-27T00:51:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1413 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:51:32 dist-test-slave-86rs krb5kdc[22207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:51:32 dist-test-slave-86rs krb5kdc[22207](info): set up 2 sockets
Sep 27 00:51:32 dist-test-slave-86rs krb5kdc[22207](info): commencing operation
krb5kdc: starting...
W20260927 00:51:33.892678 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:51:33 dist-test-slave-86rs krb5kdc[22207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:51:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:51:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:51:34.023970 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45167
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:34.141498 22220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:34.141875 22220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:34.141986 22220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:34.146647 22220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:34.146782 22220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:34.146826 22220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:34.146867 22220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:34.146906 22220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:34.152959 22220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45167
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:34.154241 22220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:34.155345 22220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:34.164551 22228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:34.164556 22225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:34.164574 22226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:34.164985 22220 server_base.cc:1034] running on GCE node
I20260927 00:51:34.165658 22220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:34.166916 22220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:34.168136 22220 hybrid_clock.cc:648] HybridClock initialized: now 1790470294168106 us; error 50 us; skew 500 ppm
Sep 27 00:51:34 dist-test-slave-86rs krb5kdc[22207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:34.171216 22220 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:51:34.172204 22220 webserver.cc:460] Webserver started at http://127.0.98.62:36637/ using document root <none> and password file <none>
I20260927 00:51:34.172956 22220 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:34.173161 22220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:34.173406 22220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:34.175596 22220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "9432d59dedf14df8ab7f05ddafb603bb"
format_stamp: "Formatted at 2026-09-27 00:51:34 on dist-test-slave-86rs"
server_key: "e558f3b98ef659236863d38f773d6942"
server_key_iv: "6947d4351aaafe86ccef443f8064acfc"
server_key_version: "encryptionkey@0"
I20260927 00:51:34.176210 22220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "9432d59dedf14df8ab7f05ddafb603bb"
format_stamp: "Formatted at 2026-09-27 00:51:34 on dist-test-slave-86rs"
server_key: "e558f3b98ef659236863d38f773d6942"
server_key_iv: "6947d4351aaafe86ccef443f8064acfc"
server_key_version: "encryptionkey@0"
I20260927 00:51:34.180024 22220 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 00:51:34.182642 22235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:34.183907 22220 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:51:34.184067 22220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "9432d59dedf14df8ab7f05ddafb603bb"
format_stamp: "Formatted at 2026-09-27 00:51:34 on dist-test-slave-86rs"
server_key: "e558f3b98ef659236863d38f773d6942"
server_key_iv: "6947d4351aaafe86ccef443f8064acfc"
server_key_version: "encryptionkey@0"
I20260927 00:51:34.184199 22220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:34.189131 22220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:34.192854 22220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:34.193094 22220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:34.200534 22220 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45167
I20260927 00:51:34.200841 22287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45167 every 8 connection(s)
I20260927 00:51:34.201579 22220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:51:34.204573 22288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:34.210590 22288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9432d59dedf14df8ab7f05ddafb603bb: Bootstrap starting.
I20260927 00:51:34.211637 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 22220
I20260927 00:51:34.211817 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:51:34.212101 392 external_mini_cluster.cc:1428] Setting key cf72d993a4dc73094249f9a55d174368
I20260927 00:51:34.213631 22288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9432d59dedf14df8ab7f05ddafb603bb: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:34.214563 22288 log.cc:824] T 00000000000000000000000000000000 P 9432d59dedf14df8ab7f05ddafb603bb: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:34.216997 22288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9432d59dedf14df8ab7f05ddafb603bb: No bootstrap required, opened a new log
I20260927 00:51:34.218014 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:34.213382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59305 (local address 127.0.98.62:45167)
0927 00:51:34.213929 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:34.213951 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:34.213997 (+ 46us) server_negotiation.cc:407] Connection header received
0927 00:51:34.214960 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:34.214985 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:34.215290 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:34.215685 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:34.216539 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:34.216545 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:34.216561 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:34.216573 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:34.216647 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:34.216676 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:34.216805 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:34.216900 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 00:51:34.217056 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":158,"threads_started":1}
I20260927 00:51:34.221331 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:51:34.221427 22288 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9432d59dedf14df8ab7f05ddafb603bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9432d59dedf14df8ab7f05ddafb603bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45167 } }
I20260927 00:51:34.221699 22288 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9432d59dedf14df8ab7f05ddafb603bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:34.221755 22288 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9432d59dedf14df8ab7f05ddafb603bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9432d59dedf14df8ab7f05ddafb603bb, State: Initialized, Role: FOLLOWER
I20260927 00:51:34.222431 22288 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9432d59dedf14df8ab7f05ddafb603bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9432d59dedf14df8ab7f05ddafb603bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45167 } }
I20260927 00:51:34.222620 22288 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9432d59dedf14df8ab7f05ddafb603bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:51:34.222687 22288 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9432d59dedf14df8ab7f05ddafb603bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:51:34.222803 22288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9432d59dedf14df8ab7f05ddafb603bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:34.224139 22288 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9432d59dedf14df8ab7f05ddafb603bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9432d59dedf14df8ab7f05ddafb603bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45167 } }
I20260927 00:51:34.224535 22288 leader_election.cc:302] T 00000000000000000000000000000000 P 9432d59dedf14df8ab7f05ddafb603bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9432d59dedf14df8ab7f05ddafb603bb; no voters:
I20260927 00:51:34.224891 22288 leader_election.cc:288] T 00000000000000000000000000000000 P 9432d59dedf14df8ab7f05ddafb603bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:51:34.224967 22298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9432d59dedf14df8ab7f05ddafb603bb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:34.225178 22298 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9432d59dedf14df8ab7f05ddafb603bb [term 1 LEADER]: Becoming Leader. State: Replica: 9432d59dedf14df8ab7f05ddafb603bb, State: Running, Role: LEADER
I20260927 00:51:34.225507 22298 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9432d59dedf14df8ab7f05ddafb603bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9432d59dedf14df8ab7f05ddafb603bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45167 } }
I20260927 00:51:34.226238 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:34.223970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36389 (local address 127.0.98.62:45167)
0927 00:51:34.224123 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:34.224127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:34.224451 (+ 324us) server_negotiation.cc:407] Connection header received
0927 00:51:34.224616 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:34.224619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:34.224685 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:34.224787 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:34.225546 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:34.225549 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:34.225551 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:34.225561 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:34.225590 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:34.225681 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:34.225757 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:34.226087 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 00:51:34.226129 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:51:34.226306 22288 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9432d59dedf14df8ab7f05ddafb603bb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:51:34.227624 22300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9432d59dedf14df8ab7f05ddafb603bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9432d59dedf14df8ab7f05ddafb603bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9432d59dedf14df8ab7f05ddafb603bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45167 } } }
I20260927 00:51:34.227622 22299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9432d59dedf14df8ab7f05ddafb603bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9432d59dedf14df8ab7f05ddafb603bb. Latest consensus state: current_term: 1 leader_uuid: "9432d59dedf14df8ab7f05ddafb603bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9432d59dedf14df8ab7f05ddafb603bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45167 } } }
I20260927 00:51:34.227788 22300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9432d59dedf14df8ab7f05ddafb603bb [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:34.227788 22299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9432d59dedf14df8ab7f05ddafb603bb [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:34.228194 22309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:51:34.231491 22309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:51:34.237262 22309 catalog_manager.cc:1347] Generated new cluster ID: 8a09b374b5ab475486417bec027b456c
I20260927 00:51:34.237365 22309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:51:34.259575 22309 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:51:34.259975 22309 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:51:34.272651 22309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9432d59dedf14df8ab7f05ddafb603bb: Generated new TSK 0
I20260927 00:51:34.273620 22309 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:51:34.287195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 22220
2026-09-27T00:51:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:51:34 dist-test-slave-86rs krb5kdc[22319](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:51:34 dist-test-slave-86rs krb5kdc[22319](info): set up 2 sockets
Sep 27 00:51:34 dist-test-slave-86rs krb5kdc[22319](info): commencing operation
krb5kdc: starting...
W20260927 00:51:35.332676 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:51:35 dist-test-slave-86rs krb5kdc[22319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:51:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:51:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:51:35.459929 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37141
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:35.574276 22332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:35.574642 22332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:35.574741 22332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:35.579094 22332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:35.579196 22332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:35.579236 22332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:35.579274 22332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:35.579313 22332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:35.584916 22332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:35.586133 22332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:35.587234 22332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:35.596304 22338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:35.596304 22340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:35.596410 22332 server_base.cc:1034] running on GCE node
W20260927 00:51:35.596607 22337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:35.597172 22332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:35.598361 22332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:35.599534 22332 hybrid_clock.cc:648] HybridClock initialized: now 1790470295599504 us; error 56 us; skew 500 ppm
Sep 27 00:51:35 dist-test-slave-86rs krb5kdc[22319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:35.602448 22332 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:51:35.603421 22332 webserver.cc:460] Webserver started at http://127.0.98.62:33673/ using document root <none> and password file <none>
I20260927 00:51:35.604029 22332 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:35.604127 22332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:35.604422 22332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:35.606611 22332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "7b671cf74c57498f8423e937dc17d6c3"
format_stamp: "Formatted at 2026-09-27 00:51:35 on dist-test-slave-86rs"
server_key: "c520dcb526266698f1069553b198f906"
server_key_iv: "5ed1a1bb67d681f00d5d39ee99af9435"
server_key_version: "encryptionkey@0"
I20260927 00:51:35.607197 22332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "7b671cf74c57498f8423e937dc17d6c3"
format_stamp: "Formatted at 2026-09-27 00:51:35 on dist-test-slave-86rs"
server_key: "c520dcb526266698f1069553b198f906"
server_key_iv: "5ed1a1bb67d681f00d5d39ee99af9435"
server_key_version: "encryptionkey@0"
I20260927 00:51:35.611158 22332 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 00:51:35.613874 22347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:35.615187 22332 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:51:35.615381 22332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "7b671cf74c57498f8423e937dc17d6c3"
format_stamp: "Formatted at 2026-09-27 00:51:35 on dist-test-slave-86rs"
server_key: "c520dcb526266698f1069553b198f906"
server_key_iv: "5ed1a1bb67d681f00d5d39ee99af9435"
server_key_version: "encryptionkey@0"
I20260927 00:51:35.615515 22332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:35.630967 22332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:35.634323 22332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:35.634547 22332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:35.641997 22332 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37141
I20260927 00:51:35.642023 22399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37141 every 8 connection(s)
I20260927 00:51:35.643026 22332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:51:35.645913 22400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:35.646697 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 22332
I20260927 00:51:35.646854 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:51:35.647132 392 external_mini_cluster.cc:1428] Setting key ef0af69f0c0c4cb2db2cbf799bb2d32c
I20260927 00:51:35.653008 22400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b671cf74c57498f8423e937dc17d6c3: Bootstrap starting.
I20260927 00:51:35.655817 22400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b671cf74c57498f8423e937dc17d6c3: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:35.656724 22400 log.cc:824] T 00000000000000000000000000000000 P 7b671cf74c57498f8423e937dc17d6c3: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:51:35 dist-test-slave-86rs krb5kdc[22319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:51:35.659078 22400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b671cf74c57498f8423e937dc17d6c3: No bootstrap required, opened a new log
I20260927 00:51:35.664997 22400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b671cf74c57498f8423e937dc17d6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b671cf74c57498f8423e937dc17d6c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37141 } }
I20260927 00:51:35.665271 22400 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b671cf74c57498f8423e937dc17d6c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:35.665326 22400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b671cf74c57498f8423e937dc17d6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b671cf74c57498f8423e937dc17d6c3, State: Initialized, Role: FOLLOWER
I20260927 00:51:35.665359 22403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:35.648655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37681 (local address 127.0.98.62:37141)
0927 00:51:35.649138 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:35.649157 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:35.649201 (+ 44us) server_negotiation.cc:407] Connection header received
0927 00:51:35.650050 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:35.650074 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:35.650529 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:35.650902 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:35.651626 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:35.653122 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:35.654103 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:35.654813 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:35.654980 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:35.658236 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:35.658243 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:35.658259 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:35.658289 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:35.660294 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:35.661639 (+ 1345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:35.661647 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:35.661654 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:35.661808 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:35.662273 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:35.662277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:35.662278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:35.662800 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:35.662983 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:35.663832 (+ 849us) server_negotiation.cc:299] Negotiation successful
0927 00:51:35.664374 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":125,"threads_started":1}
I20260927 00:51:35.665870 22400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b671cf74c57498f8423e937dc17d6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b671cf74c57498f8423e937dc17d6c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37141 } }
I20260927 00:51:35.666045 22400 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b671cf74c57498f8423e937dc17d6c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:51:35.666141 22400 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b671cf74c57498f8423e937dc17d6c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:51:35.666272 22400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b671cf74c57498f8423e937dc17d6c3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:35.667691 22400 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b671cf74c57498f8423e937dc17d6c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b671cf74c57498f8423e937dc17d6c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37141 } }
I20260927 00:51:35.668134 22400 leader_election.cc:302] T 00000000000000000000000000000000 P 7b671cf74c57498f8423e937dc17d6c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b671cf74c57498f8423e937dc17d6c3; no voters:
I20260927 00:51:35.668594 22400 leader_election.cc:288] T 00000000000000000000000000000000 P 7b671cf74c57498f8423e937dc17d6c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:51:35.668715 22405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b671cf74c57498f8423e937dc17d6c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:35.668964 22405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b671cf74c57498f8423e937dc17d6c3 [term 1 LEADER]: Becoming Leader. State: Replica: 7b671cf74c57498f8423e937dc17d6c3, State: Running, Role: LEADER
I20260927 00:51:35.669081 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:51:35.669302 22405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b671cf74c57498f8423e937dc17d6c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b671cf74c57498f8423e937dc17d6c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37141 } }
I20260927 00:51:35.670039 22400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b671cf74c57498f8423e937dc17d6c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:51:35.671253 22407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b671cf74c57498f8423e937dc17d6c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b671cf74c57498f8423e937dc17d6c3. Latest consensus state: current_term: 1 leader_uuid: "7b671cf74c57498f8423e937dc17d6c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b671cf74c57498f8423e937dc17d6c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37141 } } }
I20260927 00:51:35.671388 22407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b671cf74c57498f8423e937dc17d6c3 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:35.671649 22406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b671cf74c57498f8423e937dc17d6c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b671cf74c57498f8423e937dc17d6c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b671cf74c57498f8423e937dc17d6c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37141 } } }
I20260927 00:51:35.671739 22406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b671cf74c57498f8423e937dc17d6c3 [sys.catalog]: This master's current role is: LEADER
W20260927 00:51:35.676267 22425 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7b671cf74c57498f8423e937dc17d6c3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:51:35.676368 22425 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:51:35.676443 22415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:51:35.677412 22415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:51:35.680127 22403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:35.671367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38615 (local address 127.0.98.62:37141)
0927 00:51:35.671716 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:35.671720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:35.671737 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:51:35.671919 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:35.671922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:35.671975 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:35.672038 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:35.673161 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:35.673875 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:35.674716 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:35.674980 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:35.675096 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:35.676755 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:35.676758 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:35.676761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:35.676788 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:35.678368 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:35.678822 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:35.678825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:35.678827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:35.678916 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:35.679203 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:35.679208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:35.679210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:35.679422 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:35.679564 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:35.679817 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 00:51:35.679939 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267}
I20260927 00:51:35.684335 22415 catalog_manager.cc:1347] Generated new cluster ID: 584c78f20fc64c8288d886c7b3f100f7
I20260927 00:51:35.684424 22415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:51:35.706444 22415 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:51:35.707515 22415 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:51:35.711195 22415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b671cf74c57498f8423e937dc17d6c3: Generated new TSK 0
I20260927 00:51:35.712007 22415 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:51:35.723368 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 22332
2026-09-27T00:51:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:51:35 dist-test-slave-86rs krb5kdc[22431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:51:35 dist-test-slave-86rs krb5kdc[22431](info): set up 2 sockets
Sep 27 00:51:35 dist-test-slave-86rs krb5kdc[22431](info): commencing operation
krb5kdc: starting...
W20260927 00:51:36.780680 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:51:36 dist-test-slave-86rs krb5kdc[22431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:51:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:51:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:51:36.906774 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45655
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:37.020586 22444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:37.020949 22444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:37.021032 22444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:37.025390 22444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:37.025492 22444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:37.025530 22444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:37.025586 22444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:37.025624 22444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:37.031198 22444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:37.032541 22444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:37.033697 22444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:37.042496 22449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:37.042521 22450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:37.042773 22452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:37.042583 22444 server_base.cc:1034] running on GCE node
I20260927 00:51:37.043440 22444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:37.044672 22444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:37.045862 22444 hybrid_clock.cc:648] HybridClock initialized: now 1790470297045828 us; error 48 us; skew 500 ppm
Sep 27 00:51:37 dist-test-slave-86rs krb5kdc[22431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:37.048801 22444 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:51:37.049759 22444 webserver.cc:460] Webserver started at http://127.0.98.62:44997/ using document root <none> and password file <none>
I20260927 00:51:37.050339 22444 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:37.050410 22444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:37.050626 22444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:37.052886 22444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "ab4defbbb83b4272b85614610f645272"
format_stamp: "Formatted at 2026-09-27 00:51:37 on dist-test-slave-86rs"
server_key: "3202921f077a5c9a746574904d288773"
server_key_iv: "ef1db34edb8dab873b176285d267c3a7"
server_key_version: "encryptionkey@0"
I20260927 00:51:37.053511 22444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "ab4defbbb83b4272b85614610f645272"
format_stamp: "Formatted at 2026-09-27 00:51:37 on dist-test-slave-86rs"
server_key: "3202921f077a5c9a746574904d288773"
server_key_iv: "ef1db34edb8dab873b176285d267c3a7"
server_key_version: "encryptionkey@0"
I20260927 00:51:37.057507 22444 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 00:51:37.060403 22459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:37.061862 22444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:51:37.062045 22444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "ab4defbbb83b4272b85614610f645272"
format_stamp: "Formatted at 2026-09-27 00:51:37 on dist-test-slave-86rs"
server_key: "3202921f077a5c9a746574904d288773"
server_key_iv: "ef1db34edb8dab873b176285d267c3a7"
server_key_version: "encryptionkey@0"
I20260927 00:51:37.062203 22444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:37.075886 22444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:37.079435 22444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:37.079658 22444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:37.087235 22444 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45655
I20260927 00:51:37.087251 22511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45655 every 8 connection(s)
I20260927 00:51:37.088374 22444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:51:37.091446 22512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:37.093683 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 22444
I20260927 00:51:37.093837 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:51:37.094177 392 external_mini_cluster.cc:1428] Setting key 1828b8352d5076b05e4f5eba6702ad59
I20260927 00:51:37.099040 22512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab4defbbb83b4272b85614610f645272: Bootstrap starting.
I20260927 00:51:37.101815 22512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab4defbbb83b4272b85614610f645272: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:37.102780 22512 log.cc:824] T 00000000000000000000000000000000 P ab4defbbb83b4272b85614610f645272: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:37.104908 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:37.095828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33425 (local address 127.0.98.62:45655)
0927 00:51:37.096389 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:37.096406 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:37.096464 (+ 58us) server_negotiation.cc:407] Connection header received
0927 00:51:37.097155 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:37.097174 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:37.097557 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:37.098278 (+ 721us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:37.098745 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:37.099828 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:37.100858 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:37.101398 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:37.101615 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:37.102680 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:37.102685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:37.102701 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:37.103184 (+ 483us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:37.103288 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:37.103324 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:37.103448 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:37.103641 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 00:51:37.103894 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":137,"threads_started":1}
I20260927 00:51:37.105445 22512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab4defbbb83b4272b85614610f645272: No bootstrap required, opened a new log
I20260927 00:51:37.107995 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:51:37.109849 22512 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab4defbbb83b4272b85614610f645272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab4defbbb83b4272b85614610f645272" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45655 } }
I20260927 00:51:37.110217 22512 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab4defbbb83b4272b85614610f645272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:37.110319 22512 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab4defbbb83b4272b85614610f645272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab4defbbb83b4272b85614610f645272, State: Initialized, Role: FOLLOWER
I20260927 00:51:37.110826 22512 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab4defbbb83b4272b85614610f645272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab4defbbb83b4272b85614610f645272" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45655 } }
I20260927 00:51:37.110996 22512 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab4defbbb83b4272b85614610f645272 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:51:37.111110 22512 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab4defbbb83b4272b85614610f645272 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:51:37.111218 22512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab4defbbb83b4272b85614610f645272 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:37.112512 22512 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab4defbbb83b4272b85614610f645272 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab4defbbb83b4272b85614610f645272" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45655 } }
I20260927 00:51:37.112886 22512 leader_election.cc:302] T 00000000000000000000000000000000 P ab4defbbb83b4272b85614610f645272 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab4defbbb83b4272b85614610f645272; no voters:
I20260927 00:51:37.113276 22512 leader_election.cc:288] T 00000000000000000000000000000000 P ab4defbbb83b4272b85614610f645272 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:51:37.113410 22522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab4defbbb83b4272b85614610f645272 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:37.113657 22522 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab4defbbb83b4272b85614610f645272 [term 1 LEADER]: Becoming Leader. State: Replica: ab4defbbb83b4272b85614610f645272, State: Running, Role: LEADER
I20260927 00:51:37.114092 22522 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab4defbbb83b4272b85614610f645272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab4defbbb83b4272b85614610f645272" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45655 } }
I20260927 00:51:37.114806 22512 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab4defbbb83b4272b85614610f645272 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:51:37.115043 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:37.110442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55933 (local address 127.0.98.62:45655)
0927 00:51:37.110555 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:37.110562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:37.110770 (+ 208us) server_negotiation.cc:407] Connection header received
0927 00:51:37.110943 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:37.110947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:37.111002 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:37.111106 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:37.111406 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:37.112163 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:37.113135 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:37.113555 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:37.113667 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:37.114580 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:37.114584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:37.114587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:37.114601 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:37.114633 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:37.114645 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:37.114727 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:37.114843 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 00:51:37.114909 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 00:51:37.115563 22522 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab4defbbb83b4272b85614610f645272 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab4defbbb83b4272b85614610f645272. Latest consensus state: current_term: 1 leader_uuid: "ab4defbbb83b4272b85614610f645272" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab4defbbb83b4272b85614610f645272" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45655 } } }
I20260927 00:51:37.116372 22522 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab4defbbb83b4272b85614610f645272 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:37.116261 22524 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab4defbbb83b4272b85614610f645272 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab4defbbb83b4272b85614610f645272" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab4defbbb83b4272b85614610f645272" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45655 } } }
I20260927 00:51:37.116649 22524 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab4defbbb83b4272b85614610f645272 [sys.catalog]: This master's current role is: LEADER
W20260927 00:51:37.119828 22537 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ab4defbbb83b4272b85614610f645272: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:51:37.119905 22537 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:51:37.119968 22533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:51:37.120779 22533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:51:37.126845 22533 catalog_manager.cc:1347] Generated new cluster ID: 202ceb3a5be045b89a661b9a10cacec0
I20260927 00:51:37.126941 22533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:51:37.147030 22533 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:51:37.148452 22533 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:51:37.157676 22533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab4defbbb83b4272b85614610f645272: Generated new TSK 0
I20260927 00:51:37.158524 22533 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:51:37.173022 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 22444
2026-09-27T00:51:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1449 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:51:37 dist-test-slave-86rs krb5kdc[22543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:51:37 dist-test-slave-86rs krb5kdc[22543](info): set up 2 sockets
Sep 27 00:51:37 dist-test-slave-86rs krb5kdc[22543](info): commencing operation
krb5kdc: starting...
W20260927 00:51:38.208537 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:51:38 dist-test-slave-86rs krb5kdc[22543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:51:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:51:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:51:38.337797 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43095
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:38.456630 22556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:38.456928 22556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:38.457010 22556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:38.461199 22556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:38.461272 22556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:38.461292 22556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:38.461310 22556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:38.461326 22556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:38.467038 22556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:38.468264 22556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:38.469343 22556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:38.478268 22564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:38.478358 22561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:38.478398 22556 server_base.cc:1034] running on GCE node
W20260927 00:51:38.478268 22562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:38.479270 22556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:38.480611 22556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:38.481835 22556 hybrid_clock.cc:648] HybridClock initialized: now 1790470298481821 us; error 32 us; skew 500 ppm
Sep 27 00:51:38 dist-test-slave-86rs krb5kdc[22543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:38.485200 22556 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:51:38.486217 22556 webserver.cc:460] Webserver started at http://127.0.98.62:36741/ using document root <none> and password file <none>
I20260927 00:51:38.486847 22556 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:38.486946 22556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:38.487190 22556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:38.489424 22556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "d868c134fc134a1fba9d0dce5ca53c07"
format_stamp: "Formatted at 2026-09-27 00:51:38 on dist-test-slave-86rs"
server_key: "9350541d31dc4c61177e06b5fbb14f2f"
server_key_iv: "ecdd1af4fdb755a91a1bd408ad543cd3"
server_key_version: "encryptionkey@0"
I20260927 00:51:38.490023 22556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "d868c134fc134a1fba9d0dce5ca53c07"
format_stamp: "Formatted at 2026-09-27 00:51:38 on dist-test-slave-86rs"
server_key: "9350541d31dc4c61177e06b5fbb14f2f"
server_key_iv: "ecdd1af4fdb755a91a1bd408ad543cd3"
server_key_version: "encryptionkey@0"
I20260927 00:51:38.493984 22556 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:51:38.496712 22571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:38.498118 22556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:51:38.498294 22556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "d868c134fc134a1fba9d0dce5ca53c07"
format_stamp: "Formatted at 2026-09-27 00:51:38 on dist-test-slave-86rs"
server_key: "9350541d31dc4c61177e06b5fbb14f2f"
server_key_iv: "ecdd1af4fdb755a91a1bd408ad543cd3"
server_key_version: "encryptionkey@0"
I20260927 00:51:38.498436 22556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:38.503666 22556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:38.507208 22556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:38.507443 22556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:38.514951 22556 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43095
I20260927 00:51:38.514972 22623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43095 every 8 connection(s)
I20260927 00:51:38.515991 22556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:51:38.518999 22624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:38.524890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 22556
I20260927 00:51:38.525040 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:51:38.525103 22624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d868c134fc134a1fba9d0dce5ca53c07: Bootstrap starting.
I20260927 00:51:38.525357 392 external_mini_cluster.cc:1428] Setting key b97a7e371bf6664b3d542c9fd19b6505
I20260927 00:51:38.527828 22624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d868c134fc134a1fba9d0dce5ca53c07: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:38.528653 22624 log.cc:824] T 00000000000000000000000000000000 P d868c134fc134a1fba9d0dce5ca53c07: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:38.530876 22624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d868c134fc134a1fba9d0dce5ca53c07: No bootstrap required, opened a new log
I20260927 00:51:38.531069 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:38.526883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38667 (local address 127.0.98.62:43095)
0927 00:51:38.527366 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:38.527381 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:38.527418 (+ 37us) server_negotiation.cc:407] Connection header received
0927 00:51:38.528158 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:38.528178 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:38.528542 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:38.528892 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:38.529807 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:38.529812 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:38.529824 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:38.529835 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:38.529887 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:38.529907 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:38.530025 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:38.530125 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 00:51:38.530295 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":141,"threads_started":1}
I20260927 00:51:38.534215 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:51:38.535567 22624 raft_consensus.cc:348] T 00000000000000000000000000000000 P d868c134fc134a1fba9d0dce5ca53c07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d868c134fc134a1fba9d0dce5ca53c07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43095 } }
I20260927 00:51:38.535837 22624 raft_consensus.cc:374] T 00000000000000000000000000000000 P d868c134fc134a1fba9d0dce5ca53c07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:51:38.535931 22624 raft_consensus.cc:729] T 00000000000000000000000000000000 P d868c134fc134a1fba9d0dce5ca53c07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d868c134fc134a1fba9d0dce5ca53c07, State: Initialized, Role: FOLLOWER
I20260927 00:51:38.536525 22624 consensus_queue.cc:260] T 00000000000000000000000000000000 P d868c134fc134a1fba9d0dce5ca53c07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d868c134fc134a1fba9d0dce5ca53c07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43095 } }
I20260927 00:51:38.536712 22624 raft_consensus.cc:388] T 00000000000000000000000000000000 P d868c134fc134a1fba9d0dce5ca53c07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:51:38.536792 22624 raft_consensus.cc:482] T 00000000000000000000000000000000 P d868c134fc134a1fba9d0dce5ca53c07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:51:38.536918 22624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d868c134fc134a1fba9d0dce5ca53c07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:38.538339 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:38.536482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52133 (local address 127.0.98.62:43095)
0927 00:51:38.536611 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:38.536615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:38.536905 (+ 290us) server_negotiation.cc:407] Connection header received
0927 00:51:38.537029 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:38.537032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:38.537078 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:38.537167 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:38.537904 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:38.537908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:38.537910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:51:38.537926 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:38.537961 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:51:38.537976 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:38.538072 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:38.538135 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 00:51:38.538183 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:51:38.538340 22624 raft_consensus.cc:504] T 00000000000000000000000000000000 P d868c134fc134a1fba9d0dce5ca53c07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d868c134fc134a1fba9d0dce5ca53c07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43095 } }
I20260927 00:51:38.538758 22624 leader_election.cc:302] T 00000000000000000000000000000000 P d868c134fc134a1fba9d0dce5ca53c07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d868c134fc134a1fba9d0dce5ca53c07; no voters:
I20260927 00:51:38.539104 22624 leader_election.cc:288] T 00000000000000000000000000000000 P d868c134fc134a1fba9d0dce5ca53c07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:51:38.539212 22634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d868c134fc134a1fba9d0dce5ca53c07 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:38.539467 22634 raft_consensus.cc:686] T 00000000000000000000000000000000 P d868c134fc134a1fba9d0dce5ca53c07 [term 1 LEADER]: Becoming Leader. State: Replica: d868c134fc134a1fba9d0dce5ca53c07, State: Running, Role: LEADER
I20260927 00:51:38.539819 22634 consensus_queue.cc:237] T 00000000000000000000000000000000 P d868c134fc134a1fba9d0dce5ca53c07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d868c134fc134a1fba9d0dce5ca53c07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43095 } }
I20260927 00:51:38.540467 22624 sys_catalog.cc:564] T 00000000000000000000000000000000 P d868c134fc134a1fba9d0dce5ca53c07 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:51:38.541021 22634 sys_catalog.cc:455] T 00000000000000000000000000000000 P d868c134fc134a1fba9d0dce5ca53c07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d868c134fc134a1fba9d0dce5ca53c07. Latest consensus state: current_term: 1 leader_uuid: "d868c134fc134a1fba9d0dce5ca53c07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d868c134fc134a1fba9d0dce5ca53c07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43095 } } }
I20260927 00:51:38.541206 22634 sys_catalog.cc:458] T 00000000000000000000000000000000 P d868c134fc134a1fba9d0dce5ca53c07 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:38.541476 22635 sys_catalog.cc:455] T 00000000000000000000000000000000 P d868c134fc134a1fba9d0dce5ca53c07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d868c134fc134a1fba9d0dce5ca53c07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d868c134fc134a1fba9d0dce5ca53c07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43095 } } }
I20260927 00:51:38.541599 22635 sys_catalog.cc:458] T 00000000000000000000000000000000 P d868c134fc134a1fba9d0dce5ca53c07 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:38.541805 22642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:51:38.544840 22642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:51:38.550911 22642 catalog_manager.cc:1347] Generated new cluster ID: 8861fc10d1b94df3b12618edc9ae00fc
I20260927 00:51:38.551049 22642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:51:38.568010 22642 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:51:38.568441 22642 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:51:38.576267 22642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d868c134fc134a1fba9d0dce5ca53c07: Generated new TSK 0
I20260927 00:51:38.577135 22642 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:51:38.590378 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 22556
2026-09-27T00:51:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1417 ms)
[----------] 18 tests from AuthTokenIssuingTest (25797 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:51:38 dist-test-slave-86rs krb5kdc[22655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:51:38 dist-test-slave-86rs krb5kdc[22655](info): set up 2 sockets
Sep 27 00:51:38 dist-test-slave-86rs krb5kdc[22655](info): commencing operation
krb5kdc: starting...
W20260927 00:51:39.640712 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:51:39 dist-test-slave-86rs krb5kdc[22655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:51:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:51:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:51:39.767913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42655
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42655,127.0.98.61:34469,127.0.98.60:41835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:39.883713 22668 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:51:39.884123 22668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:39.884200 22668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:39.884303 22668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:39.888674 22668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:39.888806 22668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:39.888844 22668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:39.888886 22668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:39.888916 22668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:39.894771 22668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42655,127.0.98.61:34469,127.0.98.60:41835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:39.895942 22668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:39.897028 22668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:39.906654 22674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:39.906687 22676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:39.906654 22673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:39.906980 22668 server_base.cc:1034] running on GCE node
I20260927 00:51:39.907600 22668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:39.908862 22668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:39.910066 22668 hybrid_clock.cc:648] HybridClock initialized: now 1790470299910050 us; error 30 us; skew 500 ppm
Sep 27 00:51:39 dist-test-slave-86rs krb5kdc[22655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:39.913110 22668 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:51:39.914113 22668 webserver.cc:460] Webserver started at http://127.0.98.62:41589/ using document root <none> and password file <none>
I20260927 00:51:39.914674 22668 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:39.914737 22668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:39.914912 22668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:39.917104 22668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "ed4e40e230d140abb85fff0a54fdae00"
format_stamp: "Formatted at 2026-09-27 00:51:39 on dist-test-slave-86rs"
server_key: "36d84473f9ec3ec3d17a12dc2388d9e1"
server_key_iv: "4ff044727ff8cbf335954e530db75dbf"
server_key_version: "encryptionkey@0"
I20260927 00:51:39.917639 22668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "ed4e40e230d140abb85fff0a54fdae00"
format_stamp: "Formatted at 2026-09-27 00:51:39 on dist-test-slave-86rs"
server_key: "36d84473f9ec3ec3d17a12dc2388d9e1"
server_key_iv: "4ff044727ff8cbf335954e530db75dbf"
server_key_version: "encryptionkey@0"
I20260927 00:51:39.921450 22668 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 00:51:39.923924 22683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:39.925240 22668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:51:39.925411 22668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "ed4e40e230d140abb85fff0a54fdae00"
format_stamp: "Formatted at 2026-09-27 00:51:39 on dist-test-slave-86rs"
server_key: "36d84473f9ec3ec3d17a12dc2388d9e1"
server_key_iv: "4ff044727ff8cbf335954e530db75dbf"
server_key_version: "encryptionkey@0"
I20260927 00:51:39.925521 22668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:39.954136 22668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:39.957695 22668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:39.957919 22668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:39.965821 22668 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42655
I20260927 00:51:39.966127 22735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42655 every 8 connection(s)
I20260927 00:51:39.966923 22668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:51:39.969848 22736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:39.973740 22736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42655 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:51:39.975137 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 22668
I20260927 00:51:39.975274 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:51:39.975543 392 external_mini_cluster.cc:1428] Setting key 1cf26e59d3c614e9fb5038f609a2f3cb
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 00:51:39 dist-test-slave-86rs krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:51:39.992797 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:39.976383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46466 (local address 127.0.98.62:42655)
0927 00:51:39.976967 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:39.976987 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:39.977122 (+ 135us) server_negotiation.cc:407] Connection header received
0927 00:51:39.978231 (+ 1109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:39.978235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:39.978423 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:39.978618 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:39.979499 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:39.980488 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:39.981536 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:39.981934 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:39.982234 (+ 300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:39.987170 (+ 4936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:39.987179 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:39.987196 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:39.987232 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:39.989321 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:39.989894 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:39.989903 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:39.989909 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:39.990021 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:39.990347 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:39.990352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:39.990354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:39.990784 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:39.990937 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:39.991297 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 00:51:39.991682 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":219,"server-negotiator.queue_time_us":442,"spinlock_wait_cycles":48000,"thread_start_us":149,"threads_started":1}
I20260927 00:51:39.992797 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:39.976383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42655 (local address 127.0.0.1:46466)
0927 00:51:39.976923 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:39.976976 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:39.977900 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:39.978613 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:39.978619 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:39.979096 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:39.979345 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:39.979359 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:39.980621 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:39.980629 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:39.981403 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:39.981412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:39.982104 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:39.982108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:39.982352 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:39.983052 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:51:39.983072 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:39.986961 (+ 3889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:39.989457 (+ 2496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:39.989466 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:39.989476 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:39.989757 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:39.990125 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:39.990130 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:39.990132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:39.990249 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:39.990903 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:39.990909 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:39.991056 (+ 147us) client_negotiation.cc:769] Sending connection context
0927 00:51:39.991240 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 00:51:39.991630 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":44,"thread_start_us":155,"threads_started":1}
I20260927 00:51:39.995823 22736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:51:39.996793 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:39.996440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34469 (local address 127.0.0.1:44958)
0927 00:51:39.996573 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:39.996676 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 00:51:39.997067 22686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34469: connect: Connection refused (error 111)
I20260927 00:51:39.998157 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:39.997887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34469 (local address 127.0.0.1:44964)
0927 00:51:39.998035 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:39.998091 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 00:51:39.998347 22736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34469: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34469: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 00:51:40.004317 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34469
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42655,127.0.98.61:34469,127.0.98.60:41835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:51:40.047632 22736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } attempt: 1
I20260927 00:51:40.048677 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.048361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34469 (local address 127.0.0.1:44972)
0927 00:51:40.048531 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:40.048594 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 00:51:40.049465 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.049207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34469 (local address 127.0.0.1:44978)
0927 00:51:40.049355 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:40.049407 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260927 00:51:40.049598 22736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34469: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34469: connect: Connection refused (error 111)
I20260927 00:51:40.117798 22736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } attempt: 2
I20260927 00:51:40.118795 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.118454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34469 (local address 127.0.0.1:44982)
0927 00:51:40.118639 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:40.118706 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 00:51:40.119590 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.119357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34469 (local address 127.0.0.1:44992)
0927 00:51:40.119474 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:40.119528 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 00:51:40.119732 22736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34469: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34469: connect: Connection refused (error 111)
W20260927 00:51:40.121356 22743 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:51:40.121667 22743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:40.121691 22743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:40.121732 22743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:40.125957 22743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:40.126026 22743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:40.126047 22743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:40.126065 22743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:40.126080 22743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:40.131913 22743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42655,127.0.98.61:34469,127.0.98.60:41835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:40.133167 22743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:40.134274 22743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:40.143573 22748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:40.143604 22749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:40.143589 22751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:40.143985 22743 server_base.cc:1034] running on GCE node
I20260927 00:51:40.144743 22743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:40.146044 22743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:40.147279 22743 hybrid_clock.cc:648] HybridClock initialized: now 1790470300147260 us; error 37 us; skew 500 ppm
Sep 27 00:51:40 dist-test-slave-86rs krb5kdc[22655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:40.154865 22743 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 00:51:40.156013 22743 webserver.cc:460] Webserver started at http://127.0.98.61:41957/ using document root <none> and password file <none>
I20260927 00:51:40.156705 22743 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:40.156783 22743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:40.156975 22743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:40.159179 22743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-1/data/instance:
uuid: "9ad4f48bf43e4cb4a7c78ac9b4111f29"
format_stamp: "Formatted at 2026-09-27 00:51:40 on dist-test-slave-86rs"
server_key: "30fa4e34c50cef75442c28e92a553ab1"
server_key_iv: "98e4c342d34abfba821b1755c0d58c68"
server_key_version: "encryptionkey@0"
I20260927 00:51:40.159750 22743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-1/wal/instance:
uuid: "9ad4f48bf43e4cb4a7c78ac9b4111f29"
format_stamp: "Formatted at 2026-09-27 00:51:40 on dist-test-slave-86rs"
server_key: "30fa4e34c50cef75442c28e92a553ab1"
server_key_iv: "98e4c342d34abfba821b1755c0d58c68"
server_key_version: "encryptionkey@0"
I20260927 00:51:40.163961 22743 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:51:40.167043 22758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:40.168752 22743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:51:40.168926 22743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-1/wal
uuid: "9ad4f48bf43e4cb4a7c78ac9b4111f29"
format_stamp: "Formatted at 2026-09-27 00:51:40 on dist-test-slave-86rs"
server_key: "30fa4e34c50cef75442c28e92a553ab1"
server_key_iv: "98e4c342d34abfba821b1755c0d58c68"
server_key_version: "encryptionkey@0"
I20260927 00:51:40.169072 22743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:40.196568 22743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:40.199779 22743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:40.200004 22743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:40.207587 22743 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34469
I20260927 00:51:40.207913 22810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34469 every 8 connection(s)
I20260927 00:51:40.208695 22743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-1/data/info.pb
I20260927 00:51:40.210908 22736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } attempt: 3
I20260927 00:51:40.211521 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 22743
I20260927 00:51:40.211647 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-1/wal/instance
I20260927 00:51:40.211998 392 external_mini_cluster.cc:1428] Setting key 1ad0641eef26c55f6e0602c3007f109b
I20260927 00:51:40.212023 22811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:40.216816 22811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42655 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:51:40 dist-test-slave-86rs krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 00:51:40 dist-test-slave-86rs krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:51:40.226752 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.211530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34469 (local address 127.0.0.1:45008)
0927 00:51:40.211763 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:40.211783 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:40.211910 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:40.214683 (+ 2773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:40.214687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:40.214906 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:40.215050 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:40.215059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.217137 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.217142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:40.217947 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:40.217962 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.219143 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.219151 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:40.219286 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:40.219897 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:51:40.219914 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:40.221612 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:40.224534 (+ 2922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:40.224540 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:40.224545 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:40.224838 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:40.225206 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:40.225210 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:40.225212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:40.225306 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:40.226064 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:40.226069 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:40.226226 (+ 157us) client_negotiation.cc:769] Sending connection context
0927 00:51:40.226329 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 00:51:40.226506 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"spinlock_wait_cycles":9088}
I20260927 00:51:40.228783 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.212336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45008 (local address 127.0.98.61:34469)
0927 00:51:40.212869 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:40.212892 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:40.212943 (+ 51us) server_negotiation.cc:407] Connection header received
0927 00:51:40.213998 (+ 1055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:40.214018 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:40.214369 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:40.214740 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:40.215224 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.216959 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.218109 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.218974 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.219619 (+ 645us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:40.221828 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:40.221835 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:40.221848 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:40.221882 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:40.224409 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:40.224957 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:40.224964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:40.224971 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:40.225091 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:40.225401 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:40.225406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:40.225408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:40.225914 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:40.226996 (+ 1082us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:40.227213 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 00:51:40.227565 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":139,"threads_started":1}
I20260927 00:51:40.230770 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.218889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46480 (local address 127.0.98.62:42655)
0927 00:51:40.219020 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:40.219026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:40.219440 (+ 414us) server_negotiation.cc:407] Connection header received
0927 00:51:40.220044 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:40.220048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:40.220110 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:40.220202 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:40.220989 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.221630 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.222860 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.223112 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.223229 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:40.226159 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:40.226162 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:40.226165 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:40.226212 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:40.228513 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:40.229130 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:40.229134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:40.229136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:40.229224 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:40.229550 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:40.229555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:40.229557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:40.229755 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:40.229886 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:40.230364 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 00:51:40.230518 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51,"spinlock_wait_cycles":10624}
I20260927 00:51:40.231287 22736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:51:40.231468 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.218884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42655 (local address 127.0.0.1:46480)
0927 00:51:40.219315 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:40.219366 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:40.219909 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:40.220298 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:40.220303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:40.220732 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:40.220867 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:40.220872 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.221784 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.221787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:40.222731 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:40.222741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.223240 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.223244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:40.223408 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:40.223896 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:51:40.223907 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:40.225966 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:40.228630 (+ 2664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:40.228640 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:40.228646 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:40.228999 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:40.229363 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:40.229367 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:40.229368 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:40.229454 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:40.229883 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:40.229890 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:40.230069 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 00:51:40.230930 (+ 861us) client_negotiation.cc:240] Negotiation successful
0927 00:51:40.231173 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":180,"threads_started":1}
I20260927 00:51:40.232088 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.231828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:53670)
0927 00:51:40.231959 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:40.232018 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 00:51:40.232843 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.232556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:53674)
0927 00:51:40.232705 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:40.232757 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 00:51:40.232998 22811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } has no permanent_uuid. Determining permanent_uuid...
W20260927 00:51:40.233471 22736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41835: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
Sep 27 00:51:40 dist-test-slave-86rs krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:51:40.241778 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.233535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34469 (local address 127.0.0.1:45012)
0927 00:51:40.233670 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:40.233688 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:40.233787 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:40.234060 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:40.234063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:40.234256 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:40.234409 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:40.234417 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.235250 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.235256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:40.235826 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:40.235834 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.236397 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.236400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:40.236490 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:40.237043 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:51:40.237057 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:40.238336 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:40.240093 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:40.240095 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:40.240097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:40.240381 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:40.240667 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:40.240670 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:40.240671 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:40.240756 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:40.241253 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:40.241255 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:40.241348 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 00:51:40.241475 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 00:51:40.241614 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 00:51:40.241811 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.233747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45012 (local address 127.0.98.61:34469)
0927 00:51:40.233863 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:40.233867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:40.233887 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:51:40.233918 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:40.233920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:40.233953 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:40.234016 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:40.234559 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.235110 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.235963 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.236278 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.236398 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:40.238472 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:40.238475 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:40.238477 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:40.238500 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:40.239992 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:40.240458 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:40.240464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:40.240470 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:40.240542 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:40.240885 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:40.240889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:40.240891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:40.241149 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:40.241235 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:40.241524 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 00:51:40.241681 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 00:51:40.242702 22811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:51:40.243482 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.243188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:53682)
0927 00:51:40.243312 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:40.243411 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 00:51:40.243744 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41835
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42655,127.0.98.61:34469,127.0.98.60:41835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:40.244316 22761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
I20260927 00:51:40.245491 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.245135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:53692)
0927 00:51:40.245326 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:40.245397 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260927 00:51:40.245690 22811 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41835: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
I20260927 00:51:40.264662 22736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } attempt: 1
I20260927 00:51:40.265630 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.265317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:53700)
0927 00:51:40.265480 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:40.265550 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 00:51:40.266414 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.266200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:53710)
0927 00:51:40.266309 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:40.266356 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 00:51:40.266567 22736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41835: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
I20260927 00:51:40.295023 22811 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } attempt: 1
I20260927 00:51:40.296079 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.295754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:53718)
0927 00:51:40.295934 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:40.295999 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 00:51:40.296952 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.296748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:53726)
0927 00:51:40.296852 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:40.296898 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 00:51:40.297124 22811 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41835: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
I20260927 00:51:40.341765 22736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } attempt: 2
I20260927 00:51:40.342705 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.342386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:53734)
0927 00:51:40.342550 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:40.342623 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 00:51:40.343468 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.343246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:53740)
0927 00:51:40.343357 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:40.343408 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 00:51:40.343604 22736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41835: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
W20260927 00:51:40.362301 22818 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:51:40.362687 22818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:40.362753 22818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:40.362852 22818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 00:51:40.365319 22811 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } attempt: 2
I20260927 00:51:40.366273 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.365945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:53750)
0927 00:51:40.366127 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:40.366194 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 00:51:40.367048 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.366810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:53758)
0927 00:51:40.366935 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:40.366990 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 00:51:40.367172 22811 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41835: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
W20260927 00:51:40.367666 22818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:40.367748 22818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:40.367794 22818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:40.367833 22818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:40.367869 22818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:40.373790 22818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42655,127.0.98.61:34469,127.0.98.60:41835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:40.375046 22818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:40.376116 22818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:40.384927 22824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:40.384977 22818 server_base.cc:1034] running on GCE node
W20260927 00:51:40.384922 22826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:40.384922 22823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:40.385761 22818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:40.386962 22818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:40.388131 22818 hybrid_clock.cc:648] HybridClock initialized: now 1790470300388127 us; error 44 us; skew 500 ppm
Sep 27 00:51:40 dist-test-slave-86rs krb5kdc[22655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:40.391372 22818 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 00:51:40.392488 22818 webserver.cc:460] Webserver started at http://127.0.98.60:46397/ using document root <none> and password file <none>
I20260927 00:51:40.393101 22818 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:40.393169 22818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:40.393401 22818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:40.395581 22818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-2/data/instance:
uuid: "b834e7d793ec4759b86fd3c6ced93b11"
format_stamp: "Formatted at 2026-09-27 00:51:40 on dist-test-slave-86rs"
server_key: "4c1a51d823833eebb0383f14d86b2511"
server_key_iv: "d07225f6a684e193417e38e5b40c6b88"
server_key_version: "encryptionkey@0"
I20260927 00:51:40.396217 22818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-2/wal/instance:
uuid: "b834e7d793ec4759b86fd3c6ced93b11"
format_stamp: "Formatted at 2026-09-27 00:51:40 on dist-test-slave-86rs"
server_key: "4c1a51d823833eebb0383f14d86b2511"
server_key_iv: "d07225f6a684e193417e38e5b40c6b88"
server_key_version: "encryptionkey@0"
I20260927 00:51:40.400279 22818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:51:40.403007 22833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:40.404502 22818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:51:40.404661 22818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-2/wal
uuid: "b834e7d793ec4759b86fd3c6ced93b11"
format_stamp: "Formatted at 2026-09-27 00:51:40 on dist-test-slave-86rs"
server_key: "4c1a51d823833eebb0383f14d86b2511"
server_key_iv: "d07225f6a684e193417e38e5b40c6b88"
server_key_version: "encryptionkey@0"
I20260927 00:51:40.404799 22818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:40.441900 22818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:40.442793 22736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } attempt: 3
I20260927 00:51:40.443637 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.443347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:53772)
0927 00:51:40.443500 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:40.443559 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 00:51:40.444438 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.444173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:53776)
0927 00:51:40.444337 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:40.444383 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260927 00:51:40.444634 22736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41835: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41835: connect: Connection refused (error 111)
I20260927 00:51:40.445922 22818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:40.446133 22818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:40.453744 22818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41835
I20260927 00:51:40.454052 22885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41835 every 8 connection(s)
I20260927 00:51:40.454813 22818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-2/data/info.pb
I20260927 00:51:40.457893 22886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:40.458361 22811 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } attempt: 3
I20260927 00:51:40.461608 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 22818
I20260927 00:51:40.461747 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790470006085796-392-0/minicluster-data/master-2/wal/instance
I20260927 00:51:40.462072 392 external_mini_cluster.cc:1428] Setting key 66307bf209a914c19a12153ef2410f3b
I20260927 00:51:40.463393 22886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42655 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:51:40 dist-test-slave-86rs krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 00:51:40 dist-test-slave-86rs krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 00:51:40 dist-test-slave-86rs krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:51:40.473318 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.458965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:53788)
0927 00:51:40.459146 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:40.459166 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:40.459310 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:40.462151 (+ 2841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:40.462155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:40.462371 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:40.462494 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:40.462502 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.464279 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.464283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:40.465176 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:40.465190 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.466203 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.466207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:40.466309 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:40.466881 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:51:40.466899 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:40.468436 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:40.471221 (+ 2785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:40.471224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:40.471225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:40.471433 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:40.471803 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:40.471806 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:40.471808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:40.471902 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:40.472718 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:40.472722 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:40.472846 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 00:51:40.472965 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 00:51:40.473124 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 00:51:40.474645 22890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.462849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53794 (local address 127.0.98.60:41835)
0927 00:51:40.463163 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:40.463172 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:40.463188 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:51:40.463343 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:40.463347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:40.463390 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:40.463493 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:40.464439 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.465217 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.466361 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.467077 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.467186 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:40.470049 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:40.470058 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:40.470059 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:40.470105 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:40.471930 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:40.472511 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:40.472516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:40.472518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:40.472624 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:40.472993 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:40.472996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:40.472998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:40.473256 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:40.473437 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:40.474204 (+ 767us) server_negotiation.cc:299] Negotiation successful
0927 00:51:40.474310 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"spinlock_wait_cycles":3456,"thread_start_us":110,"threads_started":1}
I20260927 00:51:40.474648 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.459813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53788 (local address 127.0.98.60:41835)
0927 00:51:40.460383 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:40.460399 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:40.460439 (+ 40us) server_negotiation.cc:407] Connection header received
0927 00:51:40.461375 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:40.461395 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:40.461830 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:40.462221 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:40.462690 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.464104 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.465318 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.466013 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.466217 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:40.468615 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:40.468623 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:40.468639 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:40.468664 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:40.470905 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:40.471551 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:40.471558 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:40.471565 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:40.471697 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:40.471997 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:40.472002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:40.472004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:40.472586 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:40.472802 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:40.473126 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 00:51:40.473447 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"spinlock_wait_cycles":2304,"thread_start_us":149,"threads_started":1}
I20260927 00:51:40.477145 22891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.465449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42655 (local address 127.0.0.1:46484)
0927 00:51:40.465836 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:40.465888 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:40.465989 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:40.466602 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:40.466607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:40.466986 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:40.467093 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:40.467099 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.468130 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.468135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:40.469257 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:40.469273 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.470139 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.470145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:40.470289 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:40.470869 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:51:40.470918 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:40.473270 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:40.475306 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:40.475313 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:40.475318 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:40.475581 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:40.475900 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:40.475904 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:40.475905 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:40.475997 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:40.476510 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:40.476519 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:40.476651 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 00:51:40.476797 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 00:51:40.476973 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"spinlock_wait_cycles":1280,"thread_start_us":103,"threads_started":1}
I20260927 00:51:40.477480 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.466199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46484 (local address 127.0.98.62:42655)
0927 00:51:40.466326 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:40.466332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:40.466347 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:51:40.466410 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:40.466412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:40.466450 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:40.466549 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:40.467266 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.467980 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.469396 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.470022 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.470141 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:40.473437 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:40.473441 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:40.473444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:40.473475 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:40.475164 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:40.475687 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:40.475690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:40.475691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:40.475770 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:40.476106 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:40.476109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:40.476111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:40.476394 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:40.476502 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:40.477147 (+ 645us) server_negotiation.cc:299] Negotiation successful
0927 00:51:40.477288 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:51:40.478740 22886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:51:40.479667 22811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ad4f48bf43e4cb4a7c78ac9b4111f29: Bootstrap starting.
I20260927 00:51:40.482825 22811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ad4f48bf43e4cb4a7c78ac9b4111f29: Neither blocks nor log segments found. Creating new log.
Sep 27 00:51:40 dist-test-slave-86rs krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:51:40.483808 22811 log.cc:824] T 00000000000000000000000000000000 P 9ad4f48bf43e4cb4a7c78ac9b4111f29: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:51:40 dist-test-slave-86rs krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:51:40.486456 22811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ad4f48bf43e4cb4a7c78ac9b4111f29: No bootstrap required, opened a new log
I20260927 00:51:40.488361 22891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.479268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34469 (local address 127.0.0.1:45018)
0927 00:51:40.479698 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:40.479715 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:40.479836 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:40.480117 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:40.480120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:40.480331 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:40.480446 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:40.480454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.481248 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.481253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:40.482001 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:40.482012 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.482587 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.482592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:40.482703 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:40.483189 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:51:40.483210 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:40.484686 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:40.486601 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:40.486604 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:40.486606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:40.486859 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:40.487168 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:40.487172 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:40.487174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:40.487277 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:40.487739 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:40.487745 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:40.487879 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 00:51:40.487983 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 00:51:40.488120 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345}
I20260927 00:51:40.488447 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.477543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46492 (local address 127.0.98.62:42655)
0927 00:51:40.478161 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:40.478182 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:40.478197 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:51:40.478228 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:40.478234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:40.478280 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:40.478359 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:40.479142 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.479675 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.480771 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.481141 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.481310 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:40.484280 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:40.484293 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:40.484296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:40.484325 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:40.486327 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:40.486846 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:40.486850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:40.486853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:40.486954 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:40.487227 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:40.487231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:40.487233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:40.487506 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:40.487723 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:40.487939 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 00:51:40.488084 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":78,"threads_started":1}
I20260927 00:51:40.488652 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.479489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45018 (local address 127.0.98.61:34469)
0927 00:51:40.479595 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:40.479599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:40.479770 (+ 171us) server_negotiation.cc:407] Connection header received
0927 00:51:40.479948 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:40.479953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:40.480002 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:40.480081 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:40.480571 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.481132 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.482134 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.482451 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.482558 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:40.484848 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:40.484852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:40.484854 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:40.484887 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:40.486468 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:40.486970 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:40.486974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:40.486976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:40.487062 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:40.487393 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:40.487396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:40.487398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:40.487607 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:40.487691 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:40.488387 (+ 696us) server_negotiation.cc:299] Negotiation successful
0927 00:51:40.488511 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:51:40.489387 22886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:51:40.491530 22811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ad4f48bf43e4cb4a7c78ac9b4111f29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4e40e230d140abb85fff0a54fdae00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42655 } } peers { permanent_uuid: "9ad4f48bf43e4cb4a7c78ac9b4111f29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "b834e7d793ec4759b86fd3c6ced93b11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } }
I20260927 00:51:40.491824 22811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ad4f48bf43e4cb4a7c78ac9b4111f29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ad4f48bf43e4cb4a7c78ac9b4111f29, State: Initialized, Role: FOLLOWER
I20260927 00:51:40.492280 22811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ad4f48bf43e4cb4a7c78ac9b4111f29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4e40e230d140abb85fff0a54fdae00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42655 } } peers { permanent_uuid: "9ad4f48bf43e4cb4a7c78ac9b4111f29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "b834e7d793ec4759b86fd3c6ced93b11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } }
I20260927 00:51:40.494045 22811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ad4f48bf43e4cb4a7c78ac9b4111f29 [sys.catalog]: configured and running, proceeding with master startup.
Sep 27 00:51:40 dist-test-slave-86rs krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 00:51:40 dist-test-slave-86rs krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:51:40.499356 22891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.490136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:53796)
0927 00:51:40.490230 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:40.490243 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:40.490316 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:40.490605 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:40.490607 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:40.490735 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:40.490810 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:40.490815 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.491569 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.491575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:40.492214 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:40.492278 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.492811 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.492814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:40.492886 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:40.493620 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:51:40.493639 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:40.494978 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:40.497413 (+ 2435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:40.497416 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:40.497418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:40.497699 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:40.498267 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:40.498273 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:40.498278 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:40.498368 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:40.498870 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:40.498873 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:40.498953 (+ 80us) client_negotiation.cc:769] Sending connection context
0927 00:51:40.499048 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 00:51:40.499191 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 00:51:40.500068 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.490274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53796 (local address 127.0.98.60:41835)
0927 00:51:40.490390 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:40.490395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:40.490410 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:51:40.490448 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:40.490452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:40.490498 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:40.490601 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:40.490923 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.491459 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.492407 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.492694 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.492808 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:40.495150 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:40.495157 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:40.495163 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:40.495204 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:40.497292 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:40.497813 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:40.497816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:40.497818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:40.498156 (+ 338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:40.498456 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:40.498459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:40.498461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:40.498736 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:40.498825 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:40.499800 (+ 975us) server_negotiation.cc:299] Negotiation successful
0927 00:51:40.499933 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 00:51:40.500777 22905 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ad4f48bf43e4cb4a7c78ac9b4111f29: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:51:40.500914 22905 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:51:40.501122 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.489829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45026 (local address 127.0.98.61:34469)
0927 00:51:40.489944 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:40.489948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:40.489960 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:51:40.490000 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:40.490003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:40.490044 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:40.490118 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:40.490981 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.491673 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.492823 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.493115 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.493227 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:40.496317 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:40.496323 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:40.496328 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:40.496361 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:40.498727 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:40.499273 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:40.499277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:40.499278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:40.499368 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:40.499672 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:40.499675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:40.499678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:40.499888 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:40.500054 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:40.500836 (+ 782us) server_negotiation.cc:299] Negotiation successful
0927 00:51:40.500960 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 00:51:40.503446 22886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b834e7d793ec4759b86fd3c6ced93b11: Bootstrap starting.
I20260927 00:51:40.499351 22743 master_runner.cc:186] Error getting master registration for 127.0.98.62:42655: OK, instance_id { permanent_uuid: "ed4e40e230d140abb85fff0a54fdae00" instance_seqno: 1790470299963580 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:51:40.503805 22743 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:51:40.496981 22894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ad4f48bf43e4cb4a7c78ac9b4111f29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4e40e230d140abb85fff0a54fdae00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42655 } } peers { permanent_uuid: "9ad4f48bf43e4cb4a7c78ac9b4111f29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "b834e7d793ec4759b86fd3c6ced93b11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } } }
I20260927 00:51:40.503895 22894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ad4f48bf43e4cb4a7c78ac9b4111f29 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:51:40.506335 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:51:40.506567 22886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b834e7d793ec4759b86fd3c6ced93b11: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:40.507614 22886 log.cc:824] T 00000000000000000000000000000000 P b834e7d793ec4759b86fd3c6ced93b11: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:40.507752 22710 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ed4e40e230d140abb85fff0a54fdae00"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46492
I20260927 00:51:40.510327 22886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b834e7d793ec4759b86fd3c6ced93b11: No bootstrap required, opened a new log
I20260927 00:51:40.514941 22886 raft_consensus.cc:348] T 00000000000000000000000000000000 P b834e7d793ec4759b86fd3c6ced93b11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4e40e230d140abb85fff0a54fdae00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42655 } } peers { permanent_uuid: "9ad4f48bf43e4cb4a7c78ac9b4111f29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "b834e7d793ec4759b86fd3c6ced93b11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } }
I20260927 00:51:40.515254 22886 raft_consensus.cc:729] T 00000000000000000000000000000000 P b834e7d793ec4759b86fd3c6ced93b11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b834e7d793ec4759b86fd3c6ced93b11, State: Initialized, Role: FOLLOWER
I20260927 00:51:40.515625 22886 consensus_queue.cc:260] T 00000000000000000000000000000000 P b834e7d793ec4759b86fd3c6ced93b11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4e40e230d140abb85fff0a54fdae00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42655 } } peers { permanent_uuid: "9ad4f48bf43e4cb4a7c78ac9b4111f29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "b834e7d793ec4759b86fd3c6ced93b11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } }
I20260927 00:51:40.516316 22908 sys_catalog.cc:455] T 00000000000000000000000000000000 P b834e7d793ec4759b86fd3c6ced93b11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4e40e230d140abb85fff0a54fdae00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42655 } } peers { permanent_uuid: "9ad4f48bf43e4cb4a7c78ac9b4111f29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "b834e7d793ec4759b86fd3c6ced93b11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } } }
I20260927 00:51:40.516477 22908 sys_catalog.cc:458] T 00000000000000000000000000000000 P b834e7d793ec4759b86fd3c6ced93b11 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:51:40.517220 22886 sys_catalog.cc:564] T 00000000000000000000000000000000 P b834e7d793ec4759b86fd3c6ced93b11 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 00:51:40.523062 22919 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b834e7d793ec4759b86fd3c6ced93b11: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:51:40.523159 22919 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:51:40.521560 22818 master_runner.cc:186] Error getting master registration for 127.0.98.62:42655: OK, instance_id { permanent_uuid: "ed4e40e230d140abb85fff0a54fdae00" instance_seqno: 1790470299963580 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:51:40.524933 22818 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:51:40.608803 22736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } attempt: 4
Sep 27 00:51:40 dist-test-slave-86rs krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:51:40.619560 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.609473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41835 (local address 127.0.0.1:53798)
0927 00:51:40.609690 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:40.609707 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:40.609808 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:40.610147 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:40.610150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:40.610388 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:40.610508 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:40.610514 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.611532 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.611536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:40.612451 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:40.612463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.613069 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.613073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:40.613161 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:40.613747 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:51:40.613763 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:40.615433 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:40.617618 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:40.617622 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:40.617624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:40.617892 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:40.618225 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:40.618229 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:40.618232 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:40.618393 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:40.618963 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:40.618966 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:40.619085 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 00:51:40.619209 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 00:51:40.619372 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260927 00:51:40.619577 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.609560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53798 (local address 127.0.98.60:41835)
0927 00:51:40.609694 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:40.609699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:40.609770 (+ 71us) server_negotiation.cc:407] Connection header received
0927 00:51:40.609934 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:40.609937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:40.609994 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:40.610101 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:40.610643 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.611394 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.612581 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.612934 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.613052 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:40.615622 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:40.615627 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:40.615630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:40.615663 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:40.617485 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:40.618008 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:40.618012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:40.618014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:40.618112 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:40.618519 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:40.618524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:40.618525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:40.618780 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:40.618913 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:40.619282 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 00:51:40.619412 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:51:40.623795 22736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed4e40e230d140abb85fff0a54fdae00: Bootstrap starting.
I20260927 00:51:40.626780 22736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed4e40e230d140abb85fff0a54fdae00: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:40.627704 22736 log.cc:824] T 00000000000000000000000000000000 P ed4e40e230d140abb85fff0a54fdae00: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:40.630750 22736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed4e40e230d140abb85fff0a54fdae00: No bootstrap required, opened a new log
I20260927 00:51:40.634625 22736 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed4e40e230d140abb85fff0a54fdae00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4e40e230d140abb85fff0a54fdae00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42655 } } peers { permanent_uuid: "9ad4f48bf43e4cb4a7c78ac9b4111f29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "b834e7d793ec4759b86fd3c6ced93b11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } }
I20260927 00:51:40.634909 22736 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed4e40e230d140abb85fff0a54fdae00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed4e40e230d140abb85fff0a54fdae00, State: Initialized, Role: FOLLOWER
I20260927 00:51:40.635255 22736 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed4e40e230d140abb85fff0a54fdae00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4e40e230d140abb85fff0a54fdae00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42655 } } peers { permanent_uuid: "9ad4f48bf43e4cb4a7c78ac9b4111f29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "b834e7d793ec4759b86fd3c6ced93b11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } }
I20260927 00:51:40.635898 22922 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed4e40e230d140abb85fff0a54fdae00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4e40e230d140abb85fff0a54fdae00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42655 } } peers { permanent_uuid: "9ad4f48bf43e4cb4a7c78ac9b4111f29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "b834e7d793ec4759b86fd3c6ced93b11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } } }
I20260927 00:51:40.636273 22922 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed4e40e230d140abb85fff0a54fdae00 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:51:40.636760 22736 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed4e40e230d140abb85fff0a54fdae00 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:51:40.636837 22710 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed4e40e230d140abb85fff0a54fdae00 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 00:51:40.636945 22710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed4e40e230d140abb85fff0a54fdae00 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:40.638329 22710 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed4e40e230d140abb85fff0a54fdae00 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4e40e230d140abb85fff0a54fdae00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42655 } } peers { permanent_uuid: "9ad4f48bf43e4cb4a7c78ac9b4111f29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "b834e7d793ec4759b86fd3c6ced93b11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } }
I20260927 00:51:40.639336 22710 leader_election.cc:288] T 00000000000000000000000000000000 P ed4e40e230d140abb85fff0a54fdae00 [CANDIDATE]: Term 1 election: Requested vote from peers 9ad4f48bf43e4cb4a7c78ac9b4111f29 (127.0.98.61:34469), b834e7d793ec4759b86fd3c6ced93b11 (127.0.98.60:41835)
I20260927 00:51:40.640759 22785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ed4e40e230d140abb85fff0a54fdae00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9ad4f48bf43e4cb4a7c78ac9b4111f29"
I20260927 00:51:40.641014 22785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ad4f48bf43e4cb4a7c78ac9b4111f29 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:40.642377 22785 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9ad4f48bf43e4cb4a7c78ac9b4111f29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed4e40e230d140abb85fff0a54fdae00 in term 1.
I20260927 00:51:40.642907 22860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ed4e40e230d140abb85fff0a54fdae00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b834e7d793ec4759b86fd3c6ced93b11"
I20260927 00:51:40.643152 22860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b834e7d793ec4759b86fd3c6ced93b11 [term 0 FOLLOWER]: Advancing to term 1
W20260927 00:51:40.643383 22933 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ed4e40e230d140abb85fff0a54fdae00: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:51:40.643455 22933 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:51:40.644552 22860 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b834e7d793ec4759b86fd3c6ced93b11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed4e40e230d140abb85fff0a54fdae00 in term 1.
I20260927 00:51:40.648495 22686 leader_election.cc:302] T 00000000000000000000000000000000 P ed4e40e230d140abb85fff0a54fdae00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b834e7d793ec4759b86fd3c6ced93b11, ed4e40e230d140abb85fff0a54fdae00; no voters:
I20260927 00:51:40.653920 22668 master_runner.cc:263] No leader master found from master ed4e40e230d140abb85fff0a54fdae00
I20260927 00:51:40.654011 22668 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:51:40.654244 22922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed4e40e230d140abb85fff0a54fdae00 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:40.654584 22922 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed4e40e230d140abb85fff0a54fdae00 [term 1 LEADER]: Becoming Leader. State: Replica: ed4e40e230d140abb85fff0a54fdae00, State: Running, Role: LEADER
I20260927 00:51:40.654995 22922 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed4e40e230d140abb85fff0a54fdae00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4e40e230d140abb85fff0a54fdae00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42655 } } peers { permanent_uuid: "9ad4f48bf43e4cb4a7c78ac9b4111f29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "b834e7d793ec4759b86fd3c6ced93b11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } }
I20260927 00:51:40.657310 22922 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed4e40e230d140abb85fff0a54fdae00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed4e40e230d140abb85fff0a54fdae00. Latest consensus state: current_term: 1 leader_uuid: "ed4e40e230d140abb85fff0a54fdae00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4e40e230d140abb85fff0a54fdae00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42655 } } peers { permanent_uuid: "9ad4f48bf43e4cb4a7c78ac9b4111f29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "b834e7d793ec4759b86fd3c6ced93b11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } } }
I20260927 00:51:40.657542 22922 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed4e40e230d140abb85fff0a54fdae00 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:40.660755 22941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:51:40.667512 22941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:51:40.679147 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.659869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46496 (local address 127.0.98.62:42655)
0927 00:51:40.659989 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:40.659994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:40.666688 (+ 6694us) server_negotiation.cc:407] Connection header received
0927 00:51:40.668958 (+ 2270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:40.668962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:40.669026 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:40.669078 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:40.670202 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.670986 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.672202 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.672625 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.672710 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:40.675093 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:40.675100 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:40.675104 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:40.675147 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:40.677089 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:40.677642 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:40.677646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:40.677648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:40.677747 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:40.678054 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:40.678058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:40.678061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:40.678316 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:40.678425 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:40.678801 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 00:51:40.678955 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:51:40.684670 22860 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b834e7d793ec4759b86fd3c6ced93b11 [term 1 FOLLOWER]: Refusing update from remote peer ed4e40e230d140abb85fff0a54fdae00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:51:40.685578 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.659092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53814 (local address 127.0.98.60:41835)
0927 00:51:40.659226 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:40.659230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:40.661366 (+ 2136us) server_negotiation.cc:407] Connection header received
0927 00:51:40.662128 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:40.662132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:40.662199 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:40.662254 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:40.667909 (+ 5655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.668777 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.674871 (+ 6094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.675245 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.675334 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:40.681825 (+ 6491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:40.681829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:40.681833 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:40.681868 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:40.683594 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:40.684102 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:40.684106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:40.684108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:40.684196 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:40.684537 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:40.684541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:40.684543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:40.684765 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:40.684907 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:40.685263 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 00:51:40.685407 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:51:40.685635 22785 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9ad4f48bf43e4cb4a7c78ac9b4111f29 [term 1 FOLLOWER]: Refusing update from remote peer ed4e40e230d140abb85fff0a54fdae00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:51:40.686707 22922 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ed4e40e230d140abb85fff0a54fdae00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ad4f48bf43e4cb4a7c78ac9b4111f29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:51:40.688059 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:40.658280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45030 (local address 127.0.98.61:34469)
0927 00:51:40.658423 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:40.658428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:40.661129 (+ 2701us) server_negotiation.cc:407] Connection header received
0927 00:51:40.661670 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:40.661674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:40.661867 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:40.661950 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:40.670342 (+ 8392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.671127 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.676057 (+ 4930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:40.676477 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:40.676566 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:40.679336 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:40.679340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:40.679343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:40.679387 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:40.681224 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:40.681790 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:40.681794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:40.681796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:40.681904 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:40.682249 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:40.682252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:40.682254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:40.682494 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:40.682612 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:40.687745 (+ 5133us) server_negotiation.cc:299] Negotiation successful
0927 00:51:40.687892 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:51:40.688721 22922 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ed4e40e230d140abb85fff0a54fdae00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b834e7d793ec4759b86fd3c6ced93b11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:51:40.693758 22894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ad4f48bf43e4cb4a7c78ac9b4111f29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed4e40e230d140abb85fff0a54fdae00. Latest consensus state: current_term: 1 leader_uuid: "ed4e40e230d140abb85fff0a54fdae00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4e40e230d140abb85fff0a54fdae00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42655 } } peers { permanent_uuid: "9ad4f48bf43e4cb4a7c78ac9b4111f29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "b834e7d793ec4759b86fd3c6ced93b11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } } }
I20260927 00:51:40.693913 22894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ad4f48bf43e4cb4a7c78ac9b4111f29 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:51:40.694815 22908 sys_catalog.cc:455] T 00000000000000000000000000000000 P b834e7d793ec4759b86fd3c6ced93b11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed4e40e230d140abb85fff0a54fdae00. Latest consensus state: current_term: 1 leader_uuid: "ed4e40e230d140abb85fff0a54fdae00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4e40e230d140abb85fff0a54fdae00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42655 } } peers { permanent_uuid: "9ad4f48bf43e4cb4a7c78ac9b4111f29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "b834e7d793ec4759b86fd3c6ced93b11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } } }
I20260927 00:51:40.694967 22908 sys_catalog.cc:458] T 00000000000000000000000000000000 P b834e7d793ec4759b86fd3c6ced93b11 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 00:51:40.697983 22839 tablet.cc:2369] T 00000000000000000000000000000000 P b834e7d793ec4759b86fd3c6ced93b11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:51:40.698238 22939 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed4e40e230d140abb85fff0a54fdae00 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed4e40e230d140abb85fff0a54fdae00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4e40e230d140abb85fff0a54fdae00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42655 } } peers { permanent_uuid: "9ad4f48bf43e4cb4a7c78ac9b4111f29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "b834e7d793ec4759b86fd3c6ced93b11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } } }
I20260927 00:51:40.698401 22939 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed4e40e230d140abb85fff0a54fdae00 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:40.700066 22894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ad4f48bf43e4cb4a7c78ac9b4111f29 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed4e40e230d140abb85fff0a54fdae00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4e40e230d140abb85fff0a54fdae00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42655 } } peers { permanent_uuid: "9ad4f48bf43e4cb4a7c78ac9b4111f29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "b834e7d793ec4759b86fd3c6ced93b11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } } }
I20260927 00:51:40.700242 22894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ad4f48bf43e4cb4a7c78ac9b4111f29 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:51:40.702546 22946 mvcc.cc:205] Tried to move back new op lower bound from 7333766351594450944 to 7333766351487508480. Current Snapshot: MvccSnapshot[applied={T|T < 7333766351594450944}]
I20260927 00:51:40.703603 22922 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed4e40e230d140abb85fff0a54fdae00 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed4e40e230d140abb85fff0a54fdae00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4e40e230d140abb85fff0a54fdae00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42655 } } peers { permanent_uuid: "9ad4f48bf43e4cb4a7c78ac9b4111f29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "b834e7d793ec4759b86fd3c6ced93b11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } } }
I20260927 00:51:40.703742 22922 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed4e40e230d140abb85fff0a54fdae00 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:40.704984 22908 sys_catalog.cc:455] T 00000000000000000000000000000000 P b834e7d793ec4759b86fd3c6ced93b11 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed4e40e230d140abb85fff0a54fdae00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4e40e230d140abb85fff0a54fdae00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42655 } } peers { permanent_uuid: "9ad4f48bf43e4cb4a7c78ac9b4111f29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "b834e7d793ec4759b86fd3c6ced93b11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41835 } } }
I20260927 00:51:40.705120 22908 sys_catalog.cc:458] T 00000000000000000000000000000000 P b834e7d793ec4759b86fd3c6ced93b11 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:51:40.705303 22941 catalog_manager.cc:1347] Generated new cluster ID: cdedd982fb1d46d1ae8695fd43b7426b
I20260927 00:51:40.705394 22941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:51:40.712384 22948 mvcc.cc:205] Tried to move back new op lower bound from 7333766351594450944 to 7333766351487508480. Current Snapshot: MvccSnapshot[applied={T|T < 7333766351594450944}]
I20260927 00:51:40.717669 22941 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:51:40.718942 22941 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:51:40.730847 22941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed4e40e230d140abb85fff0a54fdae00: Generated new TSK 0
I20260927 00:51:40.731674 22941 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:51:40.752779 22892 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:51:40.751112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46512 (local address 127.0.98.62:42655)
0927 00:51:40.751264 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:40.751270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:40.751471 (+ 201us) server_negotiation.cc:407] Connection header received
0927 00:51:40.751720 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:40.751724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:40.751781 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:40.751844 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:40.752601 (+ 757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46512: BlockingRecv error: recv got EOF from 127.0.0.1:46512 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 00:51:40.753949 22813 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:51:40.751562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45046 (local address 127.0.98.61:34469)
0927 00:51:40.751696 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:40.751700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:40.752907 (+ 1207us) server_negotiation.cc:407] Connection header received
0927 00:51:40.753097 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:40.753100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:40.753148 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:40.753218 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:40.753800 (+ 582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45046: BlockingRecv error: recv got EOF from 127.0.0.1:45046 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 00:51:40.754563 22888 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:51:40.752300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53830 (local address 127.0.98.60:41835)
0927 00:51:40.752434 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:40.752440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:40.753473 (+ 1033us) server_negotiation.cc:407] Connection header received
0927 00:51:40.753649 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:40.753653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:40.753723 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:40.753778 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:40.754466 (+ 688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53830: BlockingRecv error: recv got EOF from 127.0.0.1:53830 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 00:51:40.758168 22813 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:51:40.757009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45050 (local address 127.0.98.61:34469)
0927 00:51:40.757141 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:40.757144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:40.757325 (+ 181us) server_negotiation.cc:407] Connection header received
0927 00:51:40.757453 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:40.757457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:40.757542 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:40.757651 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:40.758060 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45050: BlockingRecv error: recv got EOF from 127.0.0.1:45050 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 00:51:40.762401 22813 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:51:40.761535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45064 (local address 127.0.98.61:34469)
0927 00:51:40.761672 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:40.761677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:40.761691 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:51:40.761741 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:40.761744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:40.761793 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:40.761877 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:40.762283 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45064: BlockingRecv error: recv got EOF from 127.0.0.1:45064 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 00:51:40.767355 22813 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:51:40.766238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45068 (local address 127.0.98.61:34469)
0927 00:51:40.766362 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:40.766369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:40.766499 (+ 130us) server_negotiation.cc:407] Connection header received
0927 00:51:40.766618 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:40.766620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:40.766663 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:40.766740 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:40.767225 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45068: BlockingRecv error: recv got EOF from 127.0.0.1:45068 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 00:51:40.774555 22813 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:51:40.773359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45072 (local address 127.0.98.61:34469)
0927 00:51:40.773546 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:40.773551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:40.773664 (+ 113us) server_negotiation.cc:407] Connection header received
0927 00:51:40.773824 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:40.773826 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:40.773874 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:40.773967 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:40.774433 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45072: BlockingRecv error: recv got EOF from 127.0.0.1:45072 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
W20260927 00:51:40.786408 22813 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:51:40.785126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45074 (local address 127.0.98.61:34469)
0927 00:51:40.785299 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:40.785316 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:40.785476 (+ 160us) server_negotiation.cc:407] Connection header received
0927 00:51:40.785587 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:40.785590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:40.785659 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:40.785763 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:40.786263 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45074: BlockingRecv error: recv got EOF from 127.0.0.1:45074 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 00:51:40.806438 22813 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:51:40.804844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45086 (local address 127.0.98.61:34469)
0927 00:51:40.804997 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:40.805003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:40.805313 (+ 310us) server_negotiation.cc:407] Connection header received
0927 00:51:40.805493 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:40.805496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:40.805550 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:40.805632 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:40.806295 (+ 663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45086: BlockingRecv error: recv got EOF from 127.0.0.1:45086 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 00:51:40.842231 22813 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:51:40.840913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45090 (local address 127.0.98.61:34469)
0927 00:51:40.841096 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:40.841101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:40.841227 (+ 126us) server_negotiation.cc:407] Connection header received
0927 00:51:40.841425 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:40.841428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:40.841487 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:40.841583 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:40.842100 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45090: BlockingRecv error: recv got EOF from 127.0.0.1:45090 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260927 00:51:40.909864 22813 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:51:40.908535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45094 (local address 127.0.98.61:34469)
0927 00:51:40.908714 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:40.908719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:40.908875 (+ 156us) server_negotiation.cc:407] Connection header received
0927 00:51:40.909054 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:40.909057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:40.909118 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:40.909243 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:40.909732 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45094: BlockingRecv error: recv got EOF from 127.0.0.1:45094 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 00:51:41.042050 22813 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:51:41.040551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45106 (local address 127.0.98.61:34469)
0927 00:51:41.040732 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:41.040737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:41.040982 (+ 245us) server_negotiation.cc:407] Connection header received
0927 00:51:41.041196 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:41.041199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:41.041259 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:41.041357 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:41.041918 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45106: BlockingRecv error: recv got EOF from 127.0.0.1:45106 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 00:51:41.302031 22813 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:51:41.300823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38310 (local address 127.0.98.61:34469)
0927 00:51:41.300992 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:41.300997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:41.301048 (+ 51us) server_negotiation.cc:407] Connection header received
0927 00:51:41.301228 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:41.301231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:41.301292 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:41.301387 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:41.301891 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38310: BlockingRecv error: recv got EOF from 127.0.0.1:38310 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 00:51:41.502835 22905 catalog_manager.cc:1259] Loaded cluster ID: cdedd982fb1d46d1ae8695fd43b7426b
I20260927 00:51:41.503007 22905 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9ad4f48bf43e4cb4a7c78ac9b4111f29: loading cluster ID for follower catalog manager: success
I20260927 00:51:41.505117 22905 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9ad4f48bf43e4cb4a7c78ac9b4111f29: acquiring CA information for follower catalog manager: success
I20260927 00:51:41.506616 22905 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9ad4f48bf43e4cb4a7c78ac9b4111f29: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:51:41.524926 22919 catalog_manager.cc:1259] Loaded cluster ID: cdedd982fb1d46d1ae8695fd43b7426b
I20260927 00:51:41.525038 22919 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b834e7d793ec4759b86fd3c6ced93b11: loading cluster ID for follower catalog manager: success
I20260927 00:51:41.526999 22919 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b834e7d793ec4759b86fd3c6ced93b11: acquiring CA information for follower catalog manager: success
I20260927 00:51:41.528416 22919 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b834e7d793ec4759b86fd3c6ced93b11: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:51:41.821863 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:41.817032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38316 (local address 127.0.98.61:34469)
0927 00:51:41.817325 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:41.817329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:41.817349 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:51:41.817504 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:41.817508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:41.817728 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:41.817863 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:51:41.818203 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:41.818891 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:41.819970 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:41.820261 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:41.820367 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:41.820697 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:51:41.821308 (+ 611us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:51:41.821415 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:41.821534 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 00:51:41.821606 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":93,"threads_started":1}
I20260927 00:51:41.830443 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:41.825927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47814 (local address 127.0.98.60:41835)
0927 00:51:41.826220 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:41.826224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:41.826241 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:51:41.826345 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:41.826347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:41.826501 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:41.826665 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:51:41.826896 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:41.827474 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:41.828506 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:41.828850 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:41.828960 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:41.829305 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:51:41.829929 (+ 624us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:51:41.830085 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:41.830134 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 00:51:41.830199 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":96,"threads_started":1}
I20260927 00:51:41.838738 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:41.834159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38322 (local address 127.0.98.61:34469)
0927 00:51:41.834367 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:41.834373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:41.834794 (+ 421us) server_negotiation.cc:407] Connection header received
0927 00:51:41.834919 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:41.834922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:41.834988 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:41.835071 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:51:41.835468 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:41.836296 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:41.837338 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:41.837612 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:41.837714 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:41.837946 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:51:41.838066 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:51:41.838132 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:41.838466 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 00:51:41.838531 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 00:51:41.839056 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:41.834590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47828 (local address 127.0.98.60:41835)
0927 00:51:41.834729 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:41.834733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:41.835157 (+ 424us) server_negotiation.cc:407] Connection header received
0927 00:51:41.835209 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:41.835214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:41.835281 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:41.835360 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:51:41.835621 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:41.836140 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:41.837630 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:41.837966 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:41.838081 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:41.838336 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:51:41.838476 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:51:41.838558 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:41.838859 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 00:51:41.838922 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:51:41.840159 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:41.834079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54798 (local address 127.0.98.62:42655)
0927 00:51:41.834560 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:41.834564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:41.834587 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:51:41.834721 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:41.834725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:41.834893 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:41.835041 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:51:41.835275 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:41.836156 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:41.838168 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:41.838547 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:41.838654 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:41.839010 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:51:41.839550 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:51:41.839928 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:41.839979 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 00:51:41.840036 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":81,"threads_started":1}
I20260927 00:51:41.843890 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 22668
I20260927 00:51:41.852057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 22743
I20260927 00:51:41.859156 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 22818
2026-09-27T00:51:41Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3271 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:51:41 dist-test-slave-86rs krb5kdc[23034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:51:41 dist-test-slave-86rs krb5kdc[23034](info): set up 2 sockets
Sep 27 00:51:41 dist-test-slave-86rs krb5kdc[23034](info): commencing operation
krb5kdc: starting...
W20260927 00:51:42.892817 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/krb5kdc/test-user.keytab.
Sep 27 00:51:42 dist-test-slave-86rs krb5kdc[23034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:51:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:51:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:51:43.023593 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40795
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40795,127.0.98.61:46687,127.0.98.60:37805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:51:43.144639 23047 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:51:43.145068 23047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:43.145148 23047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:43.145233 23047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:51:43.149724 23047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:43.149833 23047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:43.149871 23047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:43.149921 23047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:43.149958 23047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:43.155756 23047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40795,127.0.98.61:46687,127.0.98.60:37805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:43.157110 23047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:43.158203 23047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:51:43.168030 23047 server_base.cc:1034] running on GCE node
W20260927 00:51:43.167840 23052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:43.167883 23055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:43.167840 23053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:43.169104 23047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:43.170339 23047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:43.171564 23047 hybrid_clock.cc:648] HybridClock initialized: now 1790470303171553 us; error 36 us; skew 500 ppm
Sep 27 00:51:43 dist-test-slave-86rs krb5kdc[23034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:43.174790 23047 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:51:43.175805 23047 webserver.cc:460] Webserver started at http://127.0.98.62:38227/ using document root <none> and password file <none>
I20260927 00:51:43.176513 23047 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:43.176614 23047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:43.176868 23047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:43.179163 23047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-0/data/instance:
uuid: "691f9e53e58f4e9c944bc8123ca4ec60"
format_stamp: "Formatted at 2026-09-27 00:51:43 on dist-test-slave-86rs"
server_key: "444f2358bc79ae465a3771ca3b61fe00"
server_key_iv: "7e5078803ec302ecdee5d265a7c22400"
server_key_version: "encryptionkey@0"
I20260927 00:51:43.179801 23047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-0/wal/instance:
uuid: "691f9e53e58f4e9c944bc8123ca4ec60"
format_stamp: "Formatted at 2026-09-27 00:51:43 on dist-test-slave-86rs"
server_key: "444f2358bc79ae465a3771ca3b61fe00"
server_key_iv: "7e5078803ec302ecdee5d265a7c22400"
server_key_version: "encryptionkey@0"
I20260927 00:51:43.184096 23047 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 00:51:43.186910 23062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:43.188359 23047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:51:43.188513 23047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-0/wal
uuid: "691f9e53e58f4e9c944bc8123ca4ec60"
format_stamp: "Formatted at 2026-09-27 00:51:43 on dist-test-slave-86rs"
server_key: "444f2358bc79ae465a3771ca3b61fe00"
server_key_iv: "7e5078803ec302ecdee5d265a7c22400"
server_key_version: "encryptionkey@0"
I20260927 00:51:43.188648 23047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:43.207065 23047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:43.210659 23047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:43.210918 23047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:43.218889 23047 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40795
I20260927 00:51:43.219180 23114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40795 every 8 connection(s)
I20260927 00:51:43.220018 23047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:51:43.221205 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 23047
I20260927 00:51:43.221342 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-0/wal/instance
I20260927 00:51:43.221670 392 external_mini_cluster.cc:1428] Setting key 6e6509729653846c701d5be0114bd42a
I20260927 00:51:43.223789 23115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:43.227720 23115 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40795 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 00:51:43 dist-test-slave-86rs krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:51:43.246790 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.230252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50576 (local address 127.0.98.62:40795)
0927 00:51:43.230770 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:43.230795 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:43.230981 (+ 186us) server_negotiation.cc:407] Connection header received
0927 00:51:43.232440 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:43.232464 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:43.232770 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:43.233004 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:43.234331 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.235850 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.237123 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.237858 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.238065 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:43.241203 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:43.241211 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:43.241223 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:43.241247 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:43.243217 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.243780 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.243789 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.243792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.243918 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.244379 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.244382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.244383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.244935 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:43.245064 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:43.245532 (+ 468us) server_negotiation.cc:299] Negotiation successful
0927 00:51:43.245785 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"spinlock_wait_cycles":896,"thread_start_us":150,"threads_started":1}
I20260927 00:51:43.246795 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.230252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40795 (local address 127.0.0.1:50576)
0927 00:51:43.230800 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:43.230856 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:43.231960 (+ 1104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:43.233033 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:43.233040 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:43.233650 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:43.234108 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:43.234133 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.236029 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.236032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:43.236977 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:43.236987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.238066 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.238069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:43.238191 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:43.238945 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:51:43.238965 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:43.240957 (+ 1992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:43.243354 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:43.243362 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:43.243372 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:43.243623 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:43.244026 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:43.244031 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:43.244034 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:43.244199 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:43.245073 (+ 874us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:43.245079 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:43.245270 (+ 191us) client_negotiation.cc:769] Sending connection context
0927 00:51:43.245430 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 00:51:43.245879 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":99,"spinlock_wait_cycles":71808,"thread_start_us":196,"threads_started":1}
I20260927 00:51:43.249513 23115 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46687 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:51:43.250399 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.250068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46687 (local address 127.0.0.1:48774)
0927 00:51:43.250227 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:43.250324 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 00:51:43.250594 23065 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46687: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 00:51:43.251668 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.251397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46687 (local address 127.0.0.1:48786)
0927 00:51:43.251546 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:43.251608 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 00:51:43.251888 23115 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46687: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46687: connect: Connection refused (error 111)
I20260927 00:51:43.252590 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46687
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40795,127.0.98.61:46687,127.0.98.60:37805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:51:43.301199 23115 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46687 } attempt: 1
I20260927 00:51:43.302229 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.301879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46687 (local address 127.0.0.1:48798)
0927 00:51:43.302058 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:43.302127 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 00:51:43.303109 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.302891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46687 (local address 127.0.0.1:48814)
0927 00:51:43.303002 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:43.303052 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 00:51:43.303282 23115 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46687: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46687: connect: Connection refused (error 111)
I20260927 00:51:43.371485 23115 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46687 } attempt: 2
I20260927 00:51:43.372622 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.372098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46687 (local address 127.0.0.1:48830)
0927 00:51:43.372283 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:43.372534 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 00:51:43.372135 23122 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:51:43.372922 23122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:43.372982 23122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:43.373037 23122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 00:51:43.373639 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.373386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46687 (local address 127.0.0.1:48832)
0927 00:51:43.373515 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:43.373570 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 00:51:43.373778 23115 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46687: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46687: connect: Connection refused (error 111)
W20260927 00:51:43.377812 23122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:43.377919 23122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:43.377966 23122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:43.378028 23122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:43.378067 23122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:43.383823 23122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40795,127.0.98.61:46687,127.0.98.60:37805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:43.385116 23122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:43.386185 23122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:43.395289 23130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:43.395299 23127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:43.395494 23122 server_base.cc:1034] running on GCE node
W20260927 00:51:43.395332 23128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:43.396366 23122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:43.397675 23122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:43.398895 23122 hybrid_clock.cc:648] HybridClock initialized: now 1790470303398881 us; error 29 us; skew 500 ppm
Sep 27 00:51:43 dist-test-slave-86rs krb5kdc[23034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:43.402024 23122 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 00:51:43.403003 23122 webserver.cc:460] Webserver started at http://127.0.98.61:38641/ using document root <none> and password file <none>
I20260927 00:51:43.403559 23122 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:43.403622 23122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:43.403798 23122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:43.406015 23122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-1/data/instance:
uuid: "0ed7d6209eae408cb2b168dd8f345261"
format_stamp: "Formatted at 2026-09-27 00:51:43 on dist-test-slave-86rs"
server_key: "8b10d022aac2d2c23fff7212efafdf26"
server_key_iv: "c291795bcf48cb0c7b7bd8995a7e7e6c"
server_key_version: "encryptionkey@0"
I20260927 00:51:43.406567 23122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-1/wal/instance:
uuid: "0ed7d6209eae408cb2b168dd8f345261"
format_stamp: "Formatted at 2026-09-27 00:51:43 on dist-test-slave-86rs"
server_key: "8b10d022aac2d2c23fff7212efafdf26"
server_key_iv: "c291795bcf48cb0c7b7bd8995a7e7e6c"
server_key_version: "encryptionkey@0"
I20260927 00:51:43.410532 23122 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:51:43.413149 23137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:43.414404 23122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:51:43.414538 23122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-1/wal
uuid: "0ed7d6209eae408cb2b168dd8f345261"
format_stamp: "Formatted at 2026-09-27 00:51:43 on dist-test-slave-86rs"
server_key: "8b10d022aac2d2c23fff7212efafdf26"
server_key_iv: "c291795bcf48cb0c7b7bd8995a7e7e6c"
server_key_version: "encryptionkey@0"
I20260927 00:51:43.414654 23122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:43.425843 23122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:43.428912 23122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:43.429121 23122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:43.436337 23122 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46687
I20260927 00:51:43.436642 23189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46687 every 8 connection(s)
I20260927 00:51:43.437378 23122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-1/data/info.pb
I20260927 00:51:43.439465 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 23122
I20260927 00:51:43.439615 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-1/wal/instance
I20260927 00:51:43.439935 392 external_mini_cluster.cc:1428] Setting key a13afa0880e8f8e815d55838c585f50c
I20260927 00:51:43.440630 23190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:43.444887 23190 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40795 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 00:51:43 dist-test-slave-86rs krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:51:43.460425 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.446875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50582 (local address 127.0.98.62:40795)
0927 00:51:43.447022 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:43.447026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:43.448098 (+ 1072us) server_negotiation.cc:407] Connection header received
0927 00:51:43.449109 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:43.449113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:43.449174 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:43.449277 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:43.450461 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.451289 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.452544 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.452958 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.453085 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:43.455837 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:43.455840 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:43.455842 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:43.455865 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:43.457359 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.458004 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.458007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.458009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.458128 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.458634 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.458640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.458659 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.458945 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:43.459057 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:43.459934 (+ 877us) server_negotiation.cc:299] Negotiation successful
0927 00:51:43.460094 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:51:43.461355 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.447402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40795 (local address 127.0.0.1:50582)
0927 00:51:43.447940 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:43.447983 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:43.448894 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:43.449445 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:43.449452 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:43.449786 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:43.450316 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:43.450334 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.451442 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.451445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:43.452411 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:43.452422 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.453123 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.453126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:43.453299 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:43.453941 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:51:43.453958 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:43.455646 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:43.457489 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:43.457496 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:43.457514 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:43.457889 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:43.458252 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:43.458255 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:43.458257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:43.458451 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:43.459098 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:43.459108 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:43.459598 (+ 490us) client_negotiation.cc:769] Sending connection context
0927 00:51:43.459842 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 00:51:43.460244 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":147,"threads_started":1}
I20260927 00:51:43.462852 23190 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46687 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:51:43.464943 23115 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46687 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 00:51:43.467639 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMaietF/build/debug/bin/kudu
/tmp/dist-test-taskMaietF/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37805
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40795,127.0.98.61:46687,127.0.98.60:37805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:51:43 dist-test-slave-86rs krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 00:51:43 dist-test-slave-86rs krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:51:43.475068 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.463341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46687 (local address 127.0.0.1:48836)
0927 00:51:43.463516 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:43.463526 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:43.463590 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:43.464209 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:43.464216 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:43.464449 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:43.464549 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:43.464554 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.465806 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.465809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:43.466579 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:43.466588 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.467453 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.467456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:43.467553 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:43.468085 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:51:43.468099 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:43.470274 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:43.473324 (+ 3050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:43.473327 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:43.473330 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:43.473561 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:43.473902 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:43.473905 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:43.473906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:43.474008 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:43.474519 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:43.474521 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:43.474601 (+ 80us) client_negotiation.cc:769] Sending connection context
0927 00:51:43.474728 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 00:51:43.474875 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260927 00:51:43.475081 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.463458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48836 (local address 127.0.98.61:46687)
0927 00:51:43.463800 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:43.463814 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:43.463838 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:51:43.463881 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:43.463894 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:43.464050 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:43.464259 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:43.464657 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.465709 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.466696 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.467329 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.467777 (+ 448us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:43.470421 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:43.470434 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:43.470454 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:43.470483 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:43.473208 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.473669 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.473671 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.473673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.473794 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.474120 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.474124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.474125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.474411 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:43.474563 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:43.474814 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 00:51:43.474943 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":157,"threads_started":1}
I20260927 00:51:43.475237 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.465386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46687 (local address 127.0.0.1:48844)
0927 00:51:43.465546 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:43.465565 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:43.465662 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:43.466015 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:43.466020 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:43.466249 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:43.466353 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:43.466360 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.467314 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.467318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:43.468185 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:43.468198 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.468912 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.468915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:43.469039 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:43.469518 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:51:43.469533 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:43.470854 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:43.473182 (+ 2328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:43.473188 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:43.473194 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:43.473510 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:43.473890 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:43.473894 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:43.473896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:43.474032 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:43.474583 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:43.474588 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:43.474720 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 00:51:43.474819 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 00:51:43.474992 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 00:51:43.476104 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.465447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48844 (local address 127.0.98.61:46687)
0927 00:51:43.465771 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:43.465781 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:43.465791 (+ 10us) server_negotiation.cc:407] Connection header received
0927 00:51:43.465818 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:43.465821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:43.465862 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:43.465946 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:43.466475 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.467172 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.468357 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.468766 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.469260 (+ 494us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:43.471030 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:43.471034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:43.471037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:43.471060 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:43.473038 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.473643 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.473654 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.473657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.473775 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.474138 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.474143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.474146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.474446 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:43.475608 (+ 1162us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:43.475694 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 00:51:43.475825 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":255,"thread_start_us":115,"threads_started":1}
I20260927 00:51:43.477339 23115 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37805 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:51:43.477314 23190 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37805 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:51:43.478128 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.477803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37805 (local address 127.0.0.1:43364)
0927 00:51:43.477949 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:43.478048 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 00:51:43.478180 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.477760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37805 (local address 127.0.0.1:43362)
0927 00:51:43.478018 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:43.478086 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
W20260927 00:51:43.478423 23141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37805: connect: Connection refused (error 111)
I20260927 00:51:43.478922 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.478679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37805 (local address 127.0.0.1:43366)
0927 00:51:43.478817 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:43.478867 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260927 00:51:43.479069 23115 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37805: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37805: connect: Connection refused (error 111)
I20260927 00:51:43.479708 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.479495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37805 (local address 127.0.0.1:43370)
0927 00:51:43.479601 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:43.479645 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 00:51:43.479947 23190 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37805: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37805: connect: Connection refused (error 111)
I20260927 00:51:43.510272 23115 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37805 } attempt: 1
I20260927 00:51:43.511273 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.510928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37805 (local address 127.0.0.1:43376)
0927 00:51:43.511125 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:43.511189 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 00:51:43.512076 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.511866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37805 (local address 127.0.0.1:43388)
0927 00:51:43.511971 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:43.512016 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 00:51:43.512296 23115 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37805: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37805: connect: Connection refused (error 111)
I20260927 00:51:43.529233 23190 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37805 } attempt: 1
I20260927 00:51:43.530195 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.529879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37805 (local address 127.0.0.1:43392)
0927 00:51:43.530049 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:43.530113 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 00:51:43.530983 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.530750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37805 (local address 127.0.0.1:43404)
0927 00:51:43.530876 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:43.530925 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 00:51:43.531113 23190 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37805: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37805: connect: Connection refused (error 111)
I20260927 00:51:43.587476 23115 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37805 } attempt: 2
W20260927 00:51:43.587451 23197 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:51:43.587817 23197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:51:43.587868 23197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:51:43.587934 23197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 00:51:43.588445 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.588109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37805 (local address 127.0.0.1:43412)
0927 00:51:43.588288 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:43.588361 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 00:51:43.589169 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.588953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37805 (local address 127.0.0.1:43426)
0927 00:51:43.589064 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:43.589115 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 00:51:43.589289 23115 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37805: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37805: connect: Connection refused (error 111)
W20260927 00:51:43.592442 23197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:51:43.592519 23197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:51:43.592557 23197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:51:43.592614 23197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:51:43.592665 23197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:51:43.598354 23197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40795,127.0.98.61:46687,127.0.98.60:37805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:51:43.599283 23190 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37805 } attempt: 2
I20260927 00:51:43.599628 23197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:51:43.600195 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.599867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37805 (local address 127.0.0.1:43428)
0927 00:51:43.600052 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:43.600114 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 00:51:43.601025 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.600776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37805 (local address 127.0.0.1:43442)
0927 00:51:43.600905 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:43.600967 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 00:51:43.601027 23197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:51:43.601238 23190 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37805: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37805: connect: Connection refused (error 111)
W20260927 00:51:43.609903 23206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:43.609912 23203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:51:43.609879 23204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:51:43.610230 23197 server_base.cc:1034] running on GCE node
I20260927 00:51:43.610787 23197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:51:43.611967 23197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:51:43.613145 23197 hybrid_clock.cc:648] HybridClock initialized: now 1790470303613112 us; error 50 us; skew 500 ppm
Sep 27 00:51:43 dist-test-slave-86rs krb5kdc[23034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:51:43.615922 23197 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 00:51:43.616946 23197 webserver.cc:460] Webserver started at http://127.0.98.60:46089/ using document root <none> and password file <none>
I20260927 00:51:43.617537 23197 fs_manager.cc:359] Metadata directory not provided
I20260927 00:51:43.617599 23197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:51:43.617837 23197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:51:43.620139 23197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-2/data/instance:
uuid: "61e7effd17da4661907122138da6fc4e"
format_stamp: "Formatted at 2026-09-27 00:51:43 on dist-test-slave-86rs"
server_key: "d1f7e25b909cf2c977ef2c6cd1c08b41"
server_key_iv: "be1ac145212cab8c2e9d7195bf071a2a"
server_key_version: "encryptionkey@0"
I20260927 00:51:43.620815 23197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-2/wal/instance:
uuid: "61e7effd17da4661907122138da6fc4e"
format_stamp: "Formatted at 2026-09-27 00:51:43 on dist-test-slave-86rs"
server_key: "d1f7e25b909cf2c977ef2c6cd1c08b41"
server_key_iv: "be1ac145212cab8c2e9d7195bf071a2a"
server_key_version: "encryptionkey@0"
I20260927 00:51:43.624739 23197 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:51:43.627244 23213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:51:43.628562 23197 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:51:43.628703 23197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-2/wal
uuid: "61e7effd17da4661907122138da6fc4e"
format_stamp: "Formatted at 2026-09-27 00:51:43 on dist-test-slave-86rs"
server_key: "d1f7e25b909cf2c977ef2c6cd1c08b41"
server_key_iv: "be1ac145212cab8c2e9d7195bf071a2a"
server_key_version: "encryptionkey@0"
I20260927 00:51:43.628827 23197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:51:43.658300 23197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:51:43.661811 23197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:51:43.662070 23197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:51:43.669425 23197 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37805
I20260927 00:51:43.669731 23265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37805 every 8 connection(s)
I20260927 00:51:43.670468 23197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-2/data/info.pb
I20260927 00:51:43.673345 23266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:51:43.675827 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMaietF/build/debug/bin/kudu as pid 23197
I20260927 00:51:43.676092 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMaietF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790470006085796-392-0/minicluster-data/master-2/wal/instance
I20260927 00:51:43.676468 392 external_mini_cluster.cc:1428] Setting key fbddc871bab6d8e35dc50646fbeaa16b
I20260927 00:51:43.677775 23266 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40795 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:51:43 dist-test-slave-86rs krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 00:51:43 dist-test-slave-86rs krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:51:43.688422 23115 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37805 } attempt: 3
I20260927 00:51:43.690569 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.679652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50586 (local address 127.0.98.62:40795)
0927 00:51:43.679786 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:43.679791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:43.680140 (+ 349us) server_negotiation.cc:407] Connection header received
0927 00:51:43.680399 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:43.680407 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:43.680451 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:43.680543 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:43.681408 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.681890 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.683010 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.683211 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.683349 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:43.686197 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:43.686203 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:43.686209 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:43.686270 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:43.687967 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.688626 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.688631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.688632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.688731 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.689154 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.689160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.689164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.689445 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:43.689525 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:43.690260 (+ 735us) server_negotiation.cc:299] Negotiation successful
0927 00:51:43.690399 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:51:43.691917 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.677863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43456 (local address 127.0.98.60:37805)
0927 00:51:43.678306 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:43.678323 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:43.678363 (+ 40us) server_negotiation.cc:407] Connection header received
0927 00:51:43.679043 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:43.679060 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:43.679410 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:43.679772 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:43.680687 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.681909 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.682752 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.683216 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.683425 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:43.686669 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:43.686678 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:43.686696 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:43.686728 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:43.688742 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.689315 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.689322 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.689324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.689418 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.689740 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.689743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.689745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.690076 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:43.690345 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:43.690611 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 00:51:43.691222 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":288,"thread_start_us":154,"threads_started":1}
I20260927 00:51:43.691932 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.679569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40795 (local address 127.0.0.1:50586)
0927 00:51:43.680019 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:43.680067 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:43.680273 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:43.680597 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:43.680602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:43.681050 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:43.681258 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:43.681272 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.682007 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.682011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:43.682898 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:43.682906 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.683358 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.683362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:43.683518 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:43.684333 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:51:43.684349 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:43.685974 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:43.688084 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:43.688091 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:43.688102 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:43.688492 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:43.688850 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:43.688853 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:43.688858 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:43.689034 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:43.689597 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:43.689603 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:43.689948 (+ 345us) client_negotiation.cc:769] Sending connection context
0927 00:51:43.690758 (+ 810us) client_negotiation.cc:240] Negotiation successful
0927 00:51:43.691081 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":8,"thread_start_us":132,"threads_started":1}
I20260927 00:51:43.692456 23190 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37805 } attempt: 3
I20260927 00:51:43.693364 23266 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46687 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:51:43 dist-test-slave-86rs krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 00:51:43 dist-test-slave-86rs krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 00:51:43 dist-test-slave-86rs krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 00:51:43 dist-test-slave-86rs krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:51:43.701766 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.688957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37805 (local address 127.0.0.1:43462)
0927 00:51:43.689254 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:43.689270 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:43.689337 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:43.689585 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:43.689589 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:43.689792 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:43.689904 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:43.689911 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.690905 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.690912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:43.691821 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:43.691833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.694588 (+ 2755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.694595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:43.694695 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:43.695257 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:51:43.695276 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:43.696895 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:43.698934 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:43.698937 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:43.698940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:43.699234 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:43.700175 (+ 941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:43.700182 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:43.700186 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:43.700339 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:43.701182 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:43.701186 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:43.701320 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 00:51:43.701424 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 00:51:43.701546 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211}
I20260927 00:51:43.702600 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.688991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43462 (local address 127.0.98.60:37805)
0927 00:51:43.689352 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:43.689363 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:43.689380 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:51:43.689416 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:43.689419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:43.689471 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:43.689566 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:43.690032 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.690798 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.691998 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.694462 (+ 2464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.694941 (+ 479us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:43.697033 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:43.697037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:43.697039 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:43.697062 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:43.698811 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.699922 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.699927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.699929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.700035 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.700487 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.700492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.700495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.701042 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:43.702122 (+ 1080us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:43.702224 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 00:51:43.702366 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":284,"thread_start_us":111,"threads_started":1}
I20260927 00:51:43.703823 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.692898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37805 (local address 127.0.0.1:43478)
0927 00:51:43.693062 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:43.693080 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:43.693175 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:43.693581 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:43.693585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:43.693833 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:43.693953 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:43.693961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.695458 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.695465 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:43.696467 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:43.696477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.697145 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.697148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:43.697235 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:43.698843 (+ 1608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:51:43.698862 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:43.700174 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:43.702136 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:43.702140 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:43.702142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:43.702410 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:43.702708 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:43.702711 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:43.702713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:43.702801 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:43.703280 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:43.703283 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:43.703400 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 00:51:43.703493 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 00:51:43.703632 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":1139}
I20260927 00:51:43.704511 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.693176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43478 (local address 127.0.98.60:37805)
0927 00:51:43.693309 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:43.693313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:43.693326 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:51:43.693366 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:43.693368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:43.693403 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:43.693498 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:43.694054 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.695317 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.696607 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.697017 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.697312 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:43.700372 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:43.700377 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:43.700383 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:43.700413 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:43.702011 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.702513 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.702518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.702521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.702592 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.702916 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.702919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.702922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.703113 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:43.704105 (+ 992us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:43.704179 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 00:51:43.704341 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":59}
I20260927 00:51:43.706624 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.694159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48860 (local address 127.0.98.61:46687)
0927 00:51:43.694272 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:43.694276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:43.694288 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:51:43.694329 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:43.694331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:43.694371 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:43.694442 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:43.695336 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.696089 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.697248 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.698210 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.698297 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:43.702433 (+ 4136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:43.702439 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:43.702445 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:43.702473 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:43.704847 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.705316 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.705320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.705323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.705408 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.705697 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.705700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.705702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.705941 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:43.706015 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:43.706373 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 00:51:43.706485 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 00:51:43.706940 23115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 691f9e53e58f4e9c944bc8123ca4ec60: Bootstrap starting.
I20260927 00:51:43.707086 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.693869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46687 (local address 127.0.0.1:48860)
0927 00:51:43.694009 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:43.694024 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:43.694085 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:43.694550 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:43.694556 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:43.694724 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:43.694841 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:43.694851 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.696274 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.696277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:43.697123 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:43.697130 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.698624 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.698629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:43.698777 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:43.699873 (+ 1096us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:51:43.699893 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:43.702283 (+ 2390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:43.704960 (+ 2677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:43.704966 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:43.704970 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:43.705208 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:43.705501 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:43.705506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:43.705508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:43.705593 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:43.706095 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:43.706101 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:43.706189 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 00:51:43.706797 (+ 608us) client_negotiation.cc:240] Negotiation successful
0927 00:51:43.706951 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 00:51:43.708002 23190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ed7d6209eae408cb2b168dd8f345261: Bootstrap starting.
I20260927 00:51:43.708199 23266 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37805 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:51:43.708721 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.694930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50594 (local address 127.0.98.62:40795)
0927 00:51:43.695033 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:43.695038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:43.695204 (+ 166us) server_negotiation.cc:407] Connection header received
0927 00:51:43.695380 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:43.695383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:43.695440 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:43.695492 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:43.696437 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.697232 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.698426 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.698836 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.698950 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:43.702418 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:43.702422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:43.702426 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:43.702451 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:43.706319 (+ 3868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.706919 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.706923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.706925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.707055 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.707391 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.707395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.707397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.707610 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:43.707766 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:43.708360 (+ 594us) server_negotiation.cc:299] Negotiation successful
0927 00:51:43.708507 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:51:43.710809 23115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 691f9e53e58f4e9c944bc8123ca4ec60: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:43.711625 23190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ed7d6209eae408cb2b168dd8f345261: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:43.711781 23115 log.cc:824] T 00000000000000000000000000000000 P 691f9e53e58f4e9c944bc8123ca4ec60: Log is configured to *not* fsync() on all Append() calls
I20260927 00:51:43.712582 23190 log.cc:824] T 00000000000000000000000000000000 P 0ed7d6209eae408cb2b168dd8f345261: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:51:43 dist-test-slave-86rs krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:51:43.714751 23190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ed7d6209eae408cb2b168dd8f345261: No bootstrap required, opened a new log
I20260927 00:51:43.714830 23115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 691f9e53e58f4e9c944bc8123ca4ec60: No bootstrap required, opened a new log
I20260927 00:51:43.717937 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.708753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37805 (local address 127.0.0.1:43494)
0927 00:51:43.708862 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 00:51:43.708876 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:51:43.708948 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:51:43.709231 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:51:43.709233 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:51:43.709404 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:51:43.709495 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:43.709502 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.710471 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.710473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:43.711222 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:51:43.711231 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.711854 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.711860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:51:43.712016 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:43.712573 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:51:43.712586 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:51:43.714227 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:51:43.716365 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:43.716368 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:43.716371 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:43.716624 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:43.716926 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:51:43.716929 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:51:43.716931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:51:43.717004 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:51:43.717421 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:51:43.717424 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:51:43.717507 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 00:51:43.717591 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 00:51:43.717749 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 00:51:43.718752 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.708901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43494 (local address 127.0.98.60:37805)
0927 00:51:43.709007 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:43.709013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:43.709028 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:51:43.709069 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:43.709073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:43.709116 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:43.709205 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:43.709620 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.710358 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.711355 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.711738 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.711854 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:43.714383 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:43.714387 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:43.714390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:43.714417 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:43.716211 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.716752 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.716758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.716760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.716833 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.717100 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.717103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.717105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.717308 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:43.718391 (+ 1083us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:43.718471 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 00:51:43.718587 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:51:43.718849 23190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ed7d6209eae408cb2b168dd8f345261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "691f9e53e58f4e9c944bc8123ca4ec60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40795 } } peers { permanent_uuid: "0ed7d6209eae408cb2b168dd8f345261" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46687 } } peers { permanent_uuid: "61e7effd17da4661907122138da6fc4e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37805 } }
I20260927 00:51:43.719113 23190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ed7d6209eae408cb2b168dd8f345261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed7d6209eae408cb2b168dd8f345261, State: Initialized, Role: FOLLOWER
I20260927 00:51:43.719457 23190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ed7d6209eae408cb2b168dd8f345261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "691f9e53e58f4e9c944bc8123ca4ec60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40795 } } peers { permanent_uuid: "0ed7d6209eae408cb2b168dd8f345261" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46687 } } peers { permanent_uuid: "61e7effd17da4661907122138da6fc4e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37805 } }
I20260927 00:51:43.720533 23115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 691f9e53e58f4e9c944bc8123ca4ec60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "691f9e53e58f4e9c944bc8123ca4ec60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40795 } } peers { permanent_uuid: "0ed7d6209eae408cb2b168dd8f345261" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46687 } } peers { permanent_uuid: "61e7effd17da4661907122138da6fc4e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37805 } }
I20260927 00:51:43.720847 23115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 691f9e53e58f4e9c944bc8123ca4ec60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 691f9e53e58f4e9c944bc8123ca4ec60, State: Initialized, Role: FOLLOWER
I20260927 00:51:43.721187 23190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ed7d6209eae408cb2b168dd8f345261 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:51:43.721240 23115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 691f9e53e58f4e9c944bc8123ca4ec60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "691f9e53e58f4e9c944bc8123ca4ec60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40795 } } peers { permanent_uuid: "0ed7d6209eae408cb2b168dd8f345261" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46687 } } peers { permanent_uuid: "61e7effd17da4661907122138da6fc4e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37805 } }
Sep 27 00:51:43 dist-test-slave-86rs krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790470302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:51:43.722285 23266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61e7effd17da4661907122138da6fc4e: Bootstrap starting.
I20260927 00:51:43.722769 23274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ed7d6209eae408cb2b168dd8f345261 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "691f9e53e58f4e9c944bc8123ca4ec60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40795 } } peers { permanent_uuid: "0ed7d6209eae408cb2b168dd8f345261" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46687 } } peers { permanent_uuid: "61e7effd17da4661907122138da6fc4e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37805 } } }
I20260927 00:51:43.723266 23274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ed7d6209eae408cb2b168dd8f345261 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:51:43.723577 23115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 691f9e53e58f4e9c944bc8123ca4ec60 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:51:43.725735 23266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61e7effd17da4661907122138da6fc4e: Neither blocks nor log segments found. Creating new log.
I20260927 00:51:43.726676 23266 log.cc:824] T 00000000000000000000000000000000 P 61e7effd17da4661907122138da6fc4e: Log is configured to *not* fsync() on all Append() calls
W20260927 00:51:43.727551 23286 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ed7d6209eae408cb2b168dd8f345261: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:51:43.727627 23286 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:51:43.728830 23266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61e7effd17da4661907122138da6fc4e: No bootstrap required, opened a new log
I20260927 00:51:43.726043 23122 master_runner.cc:186] Error getting master registration for 127.0.98.62:40795: OK, instance_id { permanent_uuid: "691f9e53e58f4e9c944bc8123ca4ec60" instance_seqno: 1790470303216611 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:51:43.729625 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691f9e53e58f4e9c944bc8123ca4ec60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "691f9e53e58f4e9c944bc8123ca4ec60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40795 } } peers { permanent_uuid: "0ed7d6209eae408cb2b168dd8f345261" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46687 } } peers { permanent_uuid: "61e7effd17da4661907122138da6fc4e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37805 } } }
I20260927 00:51:43.729863 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691f9e53e58f4e9c944bc8123ca4ec60 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:51:43.729354 23122 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 00:51:43.733269 23298 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 691f9e53e58f4e9c944bc8123ca4ec60: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 00:51:43.733369 23266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61e7effd17da4661907122138da6fc4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "691f9e53e58f4e9c944bc8123ca4ec60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40795 } } peers { permanent_uuid: "0ed7d6209eae408cb2b168dd8f345261" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46687 } } peers { permanent_uuid: "61e7effd17da4661907122138da6fc4e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37805 } }
I20260927 00:51:43.733654 23266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61e7effd17da4661907122138da6fc4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61e7effd17da4661907122138da6fc4e, State: Initialized, Role: FOLLOWER
W20260927 00:51:43.733932 23298 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:51:43.734028 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.710859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48874 (local address 127.0.98.61:46687)
0927 00:51:43.711174 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:43.711178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:43.711192 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:51:43.711227 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:43.711230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:43.711277 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:43.711352 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:43.716794 (+ 5442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.717557 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.718732 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.719050 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.719137 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:43.722335 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:43.722340 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:43.722343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:43.722378 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:43.724127 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.729185 (+ 5058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.729191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.729193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.729314 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.729678 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.729683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.729687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.732696 (+ 3009us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:43.732918 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:43.733731 (+ 813us) server_negotiation.cc:299] Negotiation successful
0927 00:51:43.733865 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246}
I20260927 00:51:43.734028 23266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61e7effd17da4661907122138da6fc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "691f9e53e58f4e9c944bc8123ca4ec60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40795 } } peers { permanent_uuid: "0ed7d6209eae408cb2b168dd8f345261" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46687 } } peers { permanent_uuid: "61e7effd17da4661907122138da6fc4e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37805 } }
I20260927 00:51:43.734660 23301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61e7effd17da4661907122138da6fc4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "691f9e53e58f4e9c944bc8123ca4ec60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40795 } } peers { permanent_uuid: "0ed7d6209eae408cb2b168dd8f345261" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46687 } } peers { permanent_uuid: "61e7effd17da4661907122138da6fc4e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37805 } } }
I20260927 00:51:43.734815 23301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61e7effd17da4661907122138da6fc4e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:51:43.735445 23266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61e7effd17da4661907122138da6fc4e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:51:43.733055 23047 master_runner.cc:186] Error getting master registration for 127.0.98.60:37805: OK, instance_id { permanent_uuid: "61e7effd17da4661907122138da6fc4e" instance_seqno: 1790470303667339 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:51:43.737181 23047 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:51:43.738647 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:51:43.740123 23089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "691f9e53e58f4e9c944bc8123ca4ec60"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50594
I20260927 00:51:43.740411 23089 raft_consensus.cc:482] T 00000000000000000000000000000000 P 691f9e53e58f4e9c944bc8123ca4ec60 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 00:51:43.740525 23089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 691f9e53e58f4e9c944bc8123ca4ec60 [term 0 FOLLOWER]: Advancing to term 1
W20260927 00:51:43.741492 23312 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 61e7effd17da4661907122138da6fc4e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:51:43.741567 23312 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:51:43.742013 23089 raft_consensus.cc:504] T 00000000000000000000000000000000 P 691f9e53e58f4e9c944bc8123ca4ec60 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "691f9e53e58f4e9c944bc8123ca4ec60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40795 } } peers { permanent_uuid: "0ed7d6209eae408cb2b168dd8f345261" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46687 } } peers { permanent_uuid: "61e7effd17da4661907122138da6fc4e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37805 } }
I20260927 00:51:43.743083 23089 leader_election.cc:288] T 00000000000000000000000000000000 P 691f9e53e58f4e9c944bc8123ca4ec60 [CANDIDATE]: Term 1 election: Requested vote from peers 0ed7d6209eae408cb2b168dd8f345261 (127.0.98.61:46687), 61e7effd17da4661907122138da6fc4e (127.0.98.60:37805)
I20260927 00:51:43.743877 23164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "691f9e53e58f4e9c944bc8123ca4ec60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0ed7d6209eae408cb2b168dd8f345261"
I20260927 00:51:43.744141 23164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ed7d6209eae408cb2b168dd8f345261 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:43.745667 23164 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0ed7d6209eae408cb2b168dd8f345261 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 691f9e53e58f4e9c944bc8123ca4ec60 in term 1.
I20260927 00:51:43.746860 23240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "691f9e53e58f4e9c944bc8123ca4ec60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "61e7effd17da4661907122138da6fc4e"
I20260927 00:51:43.747121 23240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61e7effd17da4661907122138da6fc4e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:51:43.748632 23240 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 61e7effd17da4661907122138da6fc4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 691f9e53e58f4e9c944bc8123ca4ec60 in term 1.
I20260927 00:51:43.749203 23065 leader_election.cc:302] T 00000000000000000000000000000000 P 691f9e53e58f4e9c944bc8123ca4ec60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed7d6209eae408cb2b168dd8f345261, 691f9e53e58f4e9c944bc8123ca4ec60; no voters:
I20260927 00:51:43.749544 23275 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 691f9e53e58f4e9c944bc8123ca4ec60 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:51:43.749789 23275 raft_consensus.cc:686] T 00000000000000000000000000000000 P 691f9e53e58f4e9c944bc8123ca4ec60 [term 1 LEADER]: Becoming Leader. State: Replica: 691f9e53e58f4e9c944bc8123ca4ec60, State: Running, Role: LEADER
I20260927 00:51:43.750160 23275 consensus_queue.cc:237] T 00000000000000000000000000000000 P 691f9e53e58f4e9c944bc8123ca4ec60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "691f9e53e58f4e9c944bc8123ca4ec60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40795 } } peers { permanent_uuid: "0ed7d6209eae408cb2b168dd8f345261" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46687 } } peers { permanent_uuid: "61e7effd17da4661907122138da6fc4e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37805 } }
I20260927 00:51:43.756138 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691f9e53e58f4e9c944bc8123ca4ec60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 691f9e53e58f4e9c944bc8123ca4ec60. Latest consensus state: current_term: 1 leader_uuid: "691f9e53e58f4e9c944bc8123ca4ec60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "691f9e53e58f4e9c944bc8123ca4ec60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40795 } } peers { permanent_uuid: "0ed7d6209eae408cb2b168dd8f345261" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46687 } } peers { permanent_uuid: "61e7effd17da4661907122138da6fc4e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37805 } } }
I20260927 00:51:43.756356 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691f9e53e58f4e9c944bc8123ca4ec60 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:43.762687 23320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:51:43.763592 23320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:51:43.769622 23240 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 61e7effd17da4661907122138da6fc4e [term 1 FOLLOWER]: Refusing update from remote peer 691f9e53e58f4e9c944bc8123ca4ec60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:51:43.770493 23275 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 691f9e53e58f4e9c944bc8123ca4ec60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e7effd17da4661907122138da6fc4e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:51:43.775204 23301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61e7effd17da4661907122138da6fc4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 691f9e53e58f4e9c944bc8123ca4ec60. Latest consensus state: current_term: 1 leader_uuid: "691f9e53e58f4e9c944bc8123ca4ec60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "691f9e53e58f4e9c944bc8123ca4ec60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40795 } } peers { permanent_uuid: "0ed7d6209eae408cb2b168dd8f345261" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46687 } } peers { permanent_uuid: "61e7effd17da4661907122138da6fc4e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37805 } } }
I20260927 00:51:43.775367 23301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61e7effd17da4661907122138da6fc4e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:51:43.776889 23164 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ed7d6209eae408cb2b168dd8f345261 [term 1 FOLLOWER]: Refusing update from remote peer 691f9e53e58f4e9c944bc8123ca4ec60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:51:43.777732 23275 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 691f9e53e58f4e9c944bc8123ca4ec60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed7d6209eae408cb2b168dd8f345261" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:51:43.782207 23274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ed7d6209eae408cb2b168dd8f345261 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 691f9e53e58f4e9c944bc8123ca4ec60. Latest consensus state: current_term: 1 leader_uuid: "691f9e53e58f4e9c944bc8123ca4ec60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "691f9e53e58f4e9c944bc8123ca4ec60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40795 } } peers { permanent_uuid: "0ed7d6209eae408cb2b168dd8f345261" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46687 } } peers { permanent_uuid: "61e7effd17da4661907122138da6fc4e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37805 } } }
I20260927 00:51:43.782382 23274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ed7d6209eae408cb2b168dd8f345261 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:51:43.788169 23314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691f9e53e58f4e9c944bc8123ca4ec60 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "691f9e53e58f4e9c944bc8123ca4ec60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "691f9e53e58f4e9c944bc8123ca4ec60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40795 } } peers { permanent_uuid: "0ed7d6209eae408cb2b168dd8f345261" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46687 } } peers { permanent_uuid: "61e7effd17da4661907122138da6fc4e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37805 } } }
I20260927 00:51:43.788401 23314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691f9e53e58f4e9c944bc8123ca4ec60 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:43.789726 23274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ed7d6209eae408cb2b168dd8f345261 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "691f9e53e58f4e9c944bc8123ca4ec60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "691f9e53e58f4e9c944bc8123ca4ec60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40795 } } peers { permanent_uuid: "0ed7d6209eae408cb2b168dd8f345261" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46687 } } peers { permanent_uuid: "61e7effd17da4661907122138da6fc4e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37805 } } }
I20260927 00:51:43.789866 23274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ed7d6209eae408cb2b168dd8f345261 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:51:43.792068 23324 mvcc.cc:205] Tried to move back new op lower bound from 7333766364231020544 to 7333766364165017600. Current Snapshot: MvccSnapshot[applied={T|T < 7333766364231020544}]
I20260927 00:51:43.794323 23327 mvcc.cc:205] Tried to move back new op lower bound from 7333766364231020544 to 7333766364165017600. Current Snapshot: MvccSnapshot[applied={T|T < 7333766364231020544}]
I20260927 00:51:43.796933 23320 catalog_manager.cc:1347] Generated new cluster ID: 16483e71ec614a50a7df176bce6c373a
I20260927 00:51:43.797362 23320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:51:43.800786 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691f9e53e58f4e9c944bc8123ca4ec60 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "691f9e53e58f4e9c944bc8123ca4ec60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "691f9e53e58f4e9c944bc8123ca4ec60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40795 } } peers { permanent_uuid: "0ed7d6209eae408cb2b168dd8f345261" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46687 } } peers { permanent_uuid: "61e7effd17da4661907122138da6fc4e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37805 } } }
I20260927 00:51:43.800956 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691f9e53e58f4e9c944bc8123ca4ec60 [sys.catalog]: This master's current role is: LEADER
I20260927 00:51:43.802631 23301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61e7effd17da4661907122138da6fc4e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "691f9e53e58f4e9c944bc8123ca4ec60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "691f9e53e58f4e9c944bc8123ca4ec60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40795 } } peers { permanent_uuid: "0ed7d6209eae408cb2b168dd8f345261" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46687 } } peers { permanent_uuid: "61e7effd17da4661907122138da6fc4e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37805 } } }
I20260927 00:51:43.802803 23301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61e7effd17da4661907122138da6fc4e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:51:43.803866 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.762299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43496 (local address 127.0.98.60:37805)
0927 00:51:43.762448 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:43.762453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:43.774736 (+ 12283us) server_negotiation.cc:407] Connection header received
0927 00:51:43.776205 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:43.776209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:43.776290 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:43.776350 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:43.781494 (+ 5144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.782279 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.785593 (+ 3314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.785980 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.786064 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:43.790281 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:43.790287 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:43.790291 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:43.790330 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:43.793049 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.800024 (+ 6975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.800029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.800031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.800139 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.801250 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.801255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.801257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.801530 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:43.801655 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:43.803551 (+ 1896us) server_negotiation.cc:299] Negotiation successful
0927 00:51:43.803697 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:51:43.805749 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.761370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50608 (local address 127.0.98.62:40795)
0927 00:51:43.761508 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:43.761516 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:43.772580 (+ 11064us) server_negotiation.cc:407] Connection header received
0927 00:51:43.773076 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:43.773080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:43.773138 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:43.773211 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:43.777379 (+ 4168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.778173 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.783514 (+ 5341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.783868 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.783960 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:43.789832 (+ 5872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:43.789836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:43.789839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:43.789873 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:43.792175 (+ 2302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.797184 (+ 5009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.797206 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.797211 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.797320 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.802006 (+ 4686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.802010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.802012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.802283 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:43.802420 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:43.805447 (+ 3027us) server_negotiation.cc:299] Negotiation successful
0927 00:51:43.805582 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:51:43.806658 23325 mvcc.cc:205] Tried to move back new op lower bound from 7333766364231020544 to 7333766364165017600. Current Snapshot: MvccSnapshot[applied={T|T < 7333766364231020544}]
I20260927 00:51:43.807528 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:43.761877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48880 (local address 127.0.98.61:46687)
0927 00:51:43.762013 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:43.762018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:43.772982 (+ 10964us) server_negotiation.cc:407] Connection header received
0927 00:51:43.774834 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:43.774839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:43.774908 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:43.774980 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:43.779467 (+ 4487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.780280 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.784600 (+ 4320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:43.784974 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:43.785058 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:43.796011 (+ 10953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:51:43.796015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:51:43.796018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:51:43.796052 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:51:43.798246 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.800756 (+ 2510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.800762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.800766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.800881 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:51:43.802762 (+ 1881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:51:43.802766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:51:43.802768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:51:43.803031 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:51:43.803170 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:43.807226 (+ 4056us) server_negotiation.cc:299] Negotiation successful
0927 00:51:43.807367 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:51:43.831943 23320 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:51:43.833772 23320 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:51:43.859019 23320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 691f9e53e58f4e9c944bc8123ca4ec60: Generated new TSK 0
I20260927 00:51:43.860172 23320 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:51:43.891384 23118 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:51:43.887922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50612 (local address 127.0.98.62:40795)
0927 00:51:43.888066 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:43.888072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:43.890118 (+ 2046us) server_negotiation.cc:407] Connection header received
0927 00:51:43.890355 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:43.890359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:43.890418 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:43.890477 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:43.891224 (+ 747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50612: BlockingRecv error: recv got EOF from 127.0.0.1:50612 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 00:51:43.893113 23269 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:51:43.888443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43504 (local address 127.0.98.60:37805)
0927 00:51:43.888562 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:43.888567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:43.891964 (+ 3397us) server_negotiation.cc:407] Connection header received
0927 00:51:43.892154 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:43.892158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:43.892263 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:43.892324 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:43.892972 (+ 648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43504: BlockingRecv error: recv got EOF from 127.0.0.1:43504 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 00:51:43.894733 23196 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:51:43.889226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48886 (local address 127.0.98.61:46687)
0927 00:51:43.889348 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:43.889353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:43.893656 (+ 4303us) server_negotiation.cc:407] Connection header received
0927 00:51:43.893919 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:43.893923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:43.893981 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:43.894037 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:43.894600 (+ 563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48886: BlockingRecv error: recv got EOF from 127.0.0.1:48886 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 00:51:43.899902 23196 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:51:43.898098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48900 (local address 127.0.98.61:46687)
0927 00:51:43.898238 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:43.898243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:43.898789 (+ 546us) server_negotiation.cc:407] Connection header received
0927 00:51:43.899029 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:43.899032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:43.899116 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:43.899174 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:43.899776 (+ 602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48900: BlockingRecv error: recv got EOF from 127.0.0.1:48900 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 00:51:43.905663 23196 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:51:43.903875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48914 (local address 127.0.98.61:46687)
0927 00:51:43.903987 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:43.903991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:43.904611 (+ 620us) server_negotiation.cc:407] Connection header received
0927 00:51:43.904822 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:43.904826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:43.904881 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:43.904937 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:43.905544 (+ 607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48914: BlockingRecv error: recv got EOF from 127.0.0.1:48914 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 00:51:43.912077 23196 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:51:43.910328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48928 (local address 127.0.98.61:46687)
0927 00:51:43.910441 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:43.910446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:43.911054 (+ 608us) server_negotiation.cc:407] Connection header received
0927 00:51:43.911276 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:43.911279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:43.911339 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:43.911395 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:43.911950 (+ 555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48928: BlockingRecv error: recv got EOF from 127.0.0.1:48928 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 00:51:43.920724 23196 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:51:43.918847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48930 (local address 127.0.98.61:46687)
0927 00:51:43.919004 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:43.919009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:43.919667 (+ 658us) server_negotiation.cc:407] Connection header received
0927 00:51:43.919881 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:43.919885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:43.919945 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:43.920001 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:43.920595 (+ 594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48930: BlockingRecv error: recv got EOF from 127.0.0.1:48930 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 00:51:43.933812 23196 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:51:43.931683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48940 (local address 127.0.98.61:46687)
0927 00:51:43.931820 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:43.931826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:43.932610 (+ 784us) server_negotiation.cc:407] Connection header received
0927 00:51:43.932842 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:43.932847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:43.932907 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:43.932963 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:43.933672 (+ 709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48940: BlockingRecv error: recv got EOF from 127.0.0.1:48940 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 00:51:43.954711 23196 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:51:43.952934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48944 (local address 127.0.98.61:46687)
0927 00:51:43.953088 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:43.953093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:43.953520 (+ 427us) server_negotiation.cc:407] Connection header received
0927 00:51:43.953762 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:43.953766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:43.953836 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:43.953898 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:43.954570 (+ 672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48944: BlockingRecv error: recv got EOF from 127.0.0.1:48944 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 00:51:43.991158 23196 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:51:43.989919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48948 (local address 127.0.98.61:46687)
0927 00:51:43.990077 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:43.990082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:43.990247 (+ 165us) server_negotiation.cc:407] Connection header received
0927 00:51:43.990371 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:43.990374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:43.990437 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:43.990544 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:43.991012 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48948: BlockingRecv error: recv got EOF from 127.0.0.1:48948 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 00:51:44.058905 23196 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:51:44.057525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48960 (local address 127.0.98.61:46687)
0927 00:51:44.057717 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:44.057721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:44.057822 (+ 101us) server_negotiation.cc:407] Connection header received
0927 00:51:44.058018 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:44.058021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:44.058077 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:44.058158 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:44.058759 (+ 601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48960: BlockingRecv error: recv got EOF from 127.0.0.1:48960 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 00:51:44.190944 23196 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:51:44.189732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48974 (local address 127.0.98.61:46687)
0927 00:51:44.189926 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:44.189930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:44.190030 (+ 100us) server_negotiation.cc:407] Connection header received
0927 00:51:44.190169 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:44.190172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:44.190223 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:44.190320 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:44.190811 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48974: BlockingRecv error: recv got EOF from 127.0.0.1:48974 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260927 00:51:44.450565 23196 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:51:44.449395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48988 (local address 127.0.98.61:46687)
0927 00:51:44.449544 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:44.449549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:44.449589 (+ 40us) server_negotiation.cc:407] Connection header received
0927 00:51:44.449743 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:44.449749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:44.449813 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:44.449904 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:51:44.450446 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48988: BlockingRecv error: recv got EOF from 127.0.0.1:48988 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 00:51:44.729085 23286 catalog_manager.cc:1259] Loaded cluster ID: 16483e71ec614a50a7df176bce6c373a
I20260927 00:51:44.729203 23286 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0ed7d6209eae408cb2b168dd8f345261: loading cluster ID for follower catalog manager: success
I20260927 00:51:44.730908 23286 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0ed7d6209eae408cb2b168dd8f345261: acquiring CA information for follower catalog manager: success
I20260927 00:51:44.732249 23286 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0ed7d6209eae408cb2b168dd8f345261: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:51:44.742869 23312 catalog_manager.cc:1259] Loaded cluster ID: 16483e71ec614a50a7df176bce6c373a
I20260927 00:51:44.742954 23312 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 61e7effd17da4661907122138da6fc4e: loading cluster ID for follower catalog manager: success
I20260927 00:51:44.744704 23312 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 61e7effd17da4661907122138da6fc4e: acquiring CA information for follower catalog manager: success
I20260927 00:51:44.745854 23312 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 61e7effd17da4661907122138da6fc4e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:51:44.969297 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:44.965032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48992 (local address 127.0.98.61:46687)
0927 00:51:44.965360 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:44.965367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:44.965386 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:51:44.965580 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:44.965583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:44.965736 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:44.965860 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:51:44.966185 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:44.966725 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:44.967634 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:44.967877 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:44.967993 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:44.968380 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:51:44.968841 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:51:44.968983 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:44.969046 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 00:51:44.969108 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":99,"threads_started":1}
I20260927 00:51:44.976158 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:44.972547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43516 (local address 127.0.98.60:37805)
0927 00:51:44.972802 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:44.972805 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:44.972820 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:51:44.972909 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:44.972911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:44.973029 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:44.973139 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:51:44.973319 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:44.973833 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:44.974666 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:44.974908 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:44.974997 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:44.975310 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:51:44.975761 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:51:44.975868 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:44.975977 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 00:51:44.976031 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":98,"threads_started":1}
I20260927 00:51:44.983731 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:44.979802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48996 (local address 127.0.98.61:46687)
0927 00:51:44.979929 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:44.979935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:44.980406 (+ 471us) server_negotiation.cc:407] Connection header received
0927 00:51:44.980570 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:44.980573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:44.980627 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:44.980700 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:51:44.980956 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:44.981638 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:44.982554 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:44.982790 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:44.982921 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:44.983187 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:51:44.983275 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:51:44.983351 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:44.983478 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 00:51:44.983539 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:51:44.984445 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:44.979796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43524 (local address 127.0.98.60:37805)
0927 00:51:44.979916 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:44.979920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:44.980079 (+ 159us) server_negotiation.cc:407] Connection header received
0927 00:51:44.980172 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:44.980175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:44.980263 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:44.980337 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:51:44.980649 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:44.981265 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:44.983009 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:44.983351 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:44.983448 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:44.983799 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:51:44.983873 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:51:44.983932 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:44.984207 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 00:51:44.984291 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:51:44.984639 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:51:44.979446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50624 (local address 127.0.98.62:40795)
0927 00:51:44.979777 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0927 00:51:44.979782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:51:44.979795 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:51:44.979914 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:51:44.979917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:51:44.980114 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:51:44.980257 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:51:44.980500 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:44.981318 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:44.983010 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:51:44.983361 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:51:44.983455 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:51:44.983766 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:51:44.984146 (+ 380us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:51:44.984265 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 00:51:44.984429 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 00:51:44.984502 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":88,"threads_started":1}
I20260927 00:51:44.988095 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 23047
I20260927 00:51:44.994915 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 23122
I20260927 00:51:45.001255 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMaietF/build/debug/bin/kudu with pid 23197
2026-09-27T00:51:45Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3140 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6412 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (298899 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST